System for control of a prosthetic device

ABSTRACT

A system for control of a prosthetic device includes at least one Inertial Measurement Unit detecting orientation of a user&#39;s foot. The at least one Inertial Measurement Unit is in communication with a device module configured to command at least one actuator of a prosthetic device. The at least one Inertial Measurement unit sends output signals related to orientation of the user&#39;s foot to the device module and the device module controls the at least one actuator of the prosthetic device based on the signals from the at least one Inertial Measurement Unit.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/706,575, filed Feb. 16, 2010, which is a continuation-in-part of U.S.patent application Ser. No. 12/027,116, filed Feb. 6, 2008, which claimspriority from U.S. Provisional Patent Application Ser. No. 60/899,834,filed Feb. 6, 2007, and U.S. Provisional Patent Application Ser. No.60/963,638, filed Aug. 6, 2007, each of which is hereby incorporated byreference in its entirety. This application also claims priority to U.S.Provisional Patent Application Ser. No. 61/168,832, filed Apr. 13, 2009,and U.S. Provisional Patent Application Ser. No. 61/221,858, filed Jun.30, 2009, each of which is also hereby incorporated by reference in itsentirety.

STATEMENT OF GOVERNMENT INTEREST

This invention was made with Government support under Contract NumberW911NF-09-C-0035 awarded by the U.S. Army RDECOM ACQ CTR. The Governmenthas certain rights in the invention.

TECHNICAL FIELD

The present invention relates to control of a prosthetic device and moreparticularly, to an apparatus and method for control of a prostheticdevice.

BACKGROUND INFORMATION

Many remote controls have been designed to manipulate robotic devices,mechanical devices, and virtual devices. There is a desire for a controlsystem that may process user signals quickly and accurately whileproviding smooth directional and proportional control of associatedobjects.

SUMMARY

In accordance with one aspect of the present invention, a prostheticdevice control system is disclosed. The control apparatus includes atleast one sensor module adapted to receive a body input signal and atleast one device module in communication with the sensor module. Thedevice module commands the prosthetic device in accordance with the atleast one body input signal.

In accordance with another aspect of the invention, the at least onedevice module includes at least one prosthetic control mode and commandsthe prosthetic device in accordance with the at least one prostheticcontrol mode. In accordance with another aspect of the presentinvention, the sensor module includes a sensor central processing unit(CPU) for processing the body input signals and a sensor communicatorfor wirelessly transmitting the body input signals to the device module.

In accordance with another aspect of the invention, the at least onesensor module includes an inertial measurement unit. In accordance witha further aspect of the present invention, the inertial measurement unitincludes at least one accelerometer. In accordance with yet a furtheraspect of the present invention, the at least one accelerometer maydetect accelerations in three perpendicular axes. In accordance withanother aspect of the present invention, the inertial measurement unitincludes at least two accelerometers and at least one gyroscope.

In accordance with another aspect of the present invention, the devicemodule detects user walking based at least on the body input signals. Inaccordance with a further aspect of the invention, the device modulestops commanding the prosthetic device when user walking is detected.

In accordance with another aspect of the present invention, the controlsystem includes a plurality of control modes including a bulk mode forcontrolling movement of an end point of the prosthetic device and afinesse mode for controlling movement of a prosthetic hand. In a furtheraspect of the invention, the finesse mode includes a plurality of gripsfor controlling movement of the prosthetic hand.

In accordance with another aspect of the present invention, the devicemodule commands the at least one prosthetic device in accordance with anorientation change. In accordance with a further aspect of the presentinvention, the device module commands the at least one prosthetic devicein accordance with a determined rate of orientation change.

In accordance with another aspect of the present invention, a method forevaluating usage of a prosthetic device includes receiving at least onefeedback signal from the prosthetic device, identifying a category forthe at least one feedback signal and recording the total duration thatthe at least one feedback signal is in each category.

These aspects of the invention are not meant to be exclusive and otherfeatures, aspects, and advantages of the present invention will bereadily apparent to those of ordinary skill in the art when read inconjunction with the appended claims and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features and advantages of the present invention will bebetter understood by reading the following detailed description, takentogether with the drawings wherein:

FIG. 1A is a schematic diagram of a prosthetic control apparatus andfunction thereof according to an embodiment of the present invention;

FIG. 1B, is a schematic diagram of another embodiment of the prostheticcontrol apparatus and function thereof of FIG. 1;

FIG. 2 is a side elevation view of one embodiment of a foot sensormodule of the control apparatus of FIG. 1 placed inside a shoe;

FIG. 3 is a side elevation of the foot sensor module of FIG. 2;

FIG. 4 is a top plan view of one embodiment of a foot sensor module;

FIG. 5A is a side plan view of a joystick according to one embodiment ofa motion reader for the foot sensor module of FIG. 2;

FIG. 5B is a side elevation view the joystick of FIG. 5A;

FIG. 6 is a cross-sectional view of the joystick of FIG. 5A;

FIG. 7A is a side plan view of a rollerball joystick according toanother embodiment of a motion reader for a foot sensor module of FIG.1A;

FIG. 7B is a top plan view of the rollerball joystick of FIG. 7A;

FIG. 8A is a top plan view of a one embodiment of a foot sensor moduleof FIG. 1A;

FIG. 8B is a top plan view of the foot sensor module of FIG. 8A, showingwhere the sensors are placed in relation to a user's foot;

FIG. 9 is a side elevation view of the foot sensor module of FIG. 8A;

FIG. 10A is a top plan view of another embodiment of a foot sensormodule;

FIG. 10B is a top plan view the foot sensor module of FIG. 10A, showingwhere the sensors are placed in relation to the user's foot;

FIG. 11A is a top plan view of yet another embodiment of a foot sensormodule;

FIG. 11B is a top plan view of the foot sensor module of FIG. 11A,showing where the sensors are placed in relation to the user's foot;

FIG. 12 is a side elevation view of the foot sensor module of FIG. 11A;

FIG. 13 is a side elevation view of the sensor module of FIG. 11B,showing where the sensors are in relation to the user's foot;

FIG. 14 is a side elevation view of yet another embodiment of a footsensor module as it is placed inside a user's shoe;

FIG. 15 is a side elevation view of yet another embodiment of a footsensor module as it is placed inside a user's shoe, showing where thesensors are in relation to a user's foot;

FIG. 16 is a side view of one embodiment of a residuum controller;

FIG. 17 is a perspective view of the residuum controller of FIG. 16;

FIG. 18 is a perspective view of the residuum controller of FIG. 16incorporated into a prosthetic support apparatus;

FIG. 19 is a side view of the residuum controller of FIG. 16 in use;

FIG. 20 is a side view of the residuum controller of FIG. 16 in use;

FIG. 21 is a side view of the residuum controller of FIG. 16 in use;

FIG. 22 is a front view of a kinematic mapping embodiment of the controlapparatus;

FIG. 23, is one method of control of a prosthetic device;

FIG. 24 is the method of control of the prosthetic device according toFIG. 23 with an additional holding step;

FIG. 25 is a schematic diagram of a control method during a setup state;

FIG. 26 is a schematic diagram of a control method during a deactivatedstate;

FIG. 27 is a schematic diagram of a control method during an activatedstate;

FIG. 28 is a side view of a foot sensor module according to yet anotherembodiment of the present invention;

FIG. 29 is a top view of a sensor grid of the foot sensor module of FIG.28;

FIG. 30 is a top view of a pressure profile generated by the sensor gridof FIG. 29;

FIG. 31A is a schematic diagram of a prosthetic control apparatusaccording to another embodiment of the present invention;

FIG. 31B is another embodiment of the prosthetic control apparatus ofFIG. 31A;

FIG. 32 is a front perspective view of two sensor modules of FIG. 31Bbeing used by a user;

FIG. 33 is a exploded perspective view of a housing for an inertialmeasurement unit according to an embodiment of the present invention;

FIG. 34 is a partially exploded view of an inertial measurement unitaccording to an embodiment of the present invention;

FIG. 35 is a side view of an inertial measurement unit of FIG. 32 tiltedforward;

FIG. 36 is a front view of an inertial measurement unit of FIG. 32tilted sideways;

FIG. 37 is side view of the inertial measurement unit of FIG. 35;

FIG. 38 is a front view of the inertial measurement unit of FIG. 36;

FIG. 39 is a top view of an inertial measurement unit of FIG. 32;

FIG. 40 is a side perspective view of an inertial measurement unitaccording to an embodiment of the present invention;

FIG. 41 is a process diagram of an embodiment for walk detectionaccording to the present invention;

FIG. 42 is a process diagram of an embodiment for mode changingaccording to the present invention;

FIG. 43 is a side perspective view of an embodiment of bulk controlaccording to the present invention;

FIG. 44 is a side perspective view of another embodiment of bulk controlaccording to the present invention;

FIG. 45 is side view of the bulk control of FIG. 44;

FIG. 46 is a side perspective view of a control mode according to anembodiment of the present invention;

FIG. 47 is an enlarged perspective view of a finesse control modeaccording to an embodiment of the present invention;

FIG. 48 is a side view of another embodiment of a finesse mode;

FIGS. 49A-49D are an embodiment of a finesse mode grip according to thepresent invention;

FIGS. 50A and 50B are another embodiment of a finesse mode gripaccording to the present invention;

FIG. 51 is another embodiment of a finesse mode grip according to thepresent invention;

FIG. 52 is another embodiment of a finesse mode grip according to thepresent invention;

FIG. 53 is another embodiment of a finesse mode grip according to thepresent invention;

FIG. 54 is another embodiment of a finesse mode grip according to thepresent invention;

FIG. 55 is a top view of another embodiment of a sensor module accordingto the present invention;

FIG. 56 is a side view of the sensor module of FIG. 55;

FIG. 57 is a front view of the sensor module of FIG. 55;

FIG. 58 is a process diagram of an embodiment of data collectionaccording to the present invention;

FIG. 59 is a histogram of data collected according to the datacollection embodiment of FIG. 58;

FIG. 60 is a histogram of data collected according to the datacollection embodiment of FIG. 58; and

FIG. 61 is a histogram of data collected according to the datacollection embodiment of FIG. 58.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring to FIG. 1A, a schematic view of a control apparatus 10 for aprosthetic device 12 is shown. The prosthetic device 12 includes aplurality of actuators 13 that control movement of the prosthetic device12 and one or more feedback sensors 14 for detecting the status of theprosthetic device 12 and/or support (not shown) for the prostheticdevice 12. The control apparatus 10 comprises a sensor module 15 fordetecting body input 16 and a device module 17 for commanding theprosthetic device 12. The control apparatus 10 may be used to control avariety of prosthetic devices, such as those disclosed in U.S. patentapplication Ser. No. 12/027,141, filed Feb. 6, 2008, and the U.S. patentapplication entitled ARM PROSTHETIC DEVICE, filed on the same day as thepresent application and assigned to the same assignee, each of which ishereby incorporated by reference in its entirety.

The sensor module 15 includes one or more sensors 18 connected to asensor central processing unit (sensor CPU) 19 that is connected to asensor module communicator 20. The one or more sensors 18 may bedisposed at various locations on a user to sense the body input 16 fromthe user. For example, the sensor 18 may be located to provide pressureinformation supplied by a foot 21 of the user. Similarly, sensors 18 maybe positioned to measure body input 16 from other body parts of the usersuch as a head 22, an upper torso 23, a waist 24 or a shoulder 25. Invarious embodiments, sensors 18 may measure, but are not limited to, oneor more of the following: orientation, pressure, force, rate, oracceleration. Alternatively, in some embodiments, the sensors 18 may beEMG electrodes. In some embodiments, EMG electrode signals may be usedin various controls, for example, but not limited to, turn on shouldercontrol, grip change control or movement control. The sensor CPU 19inputs data from the one or more sensors 18 and filters and/or convertsthe data to generate user input signals. The user input signals are thensent to the device module 17 by the sensor module communicator 20. Thesensor module communicator 20 may be hard wired to the device module 17or may transmit the user input signals wirelessly, for example, but notlimited to, through a radio transmitter, Bluetooth® or the like.

In some embodiments, the device module 17 includes a device CPU 26connected to a prosthetic controller 27. The device CPU 26 receives theuser input signals from the sensor module communicator 20 and prostheticdevice status signals from the feedback sensors 14. Based on the signalsfrom the sensor module communicator 20 and the feedback sensor 14, thedevice CPU 26 calculates prosthetic device actuator commands that aresent to the prosthetic actuators 13 by the prosthetic controller 27 tocommand movement of the prosthetic device 12.

Although shown as having a separate sensor module 15 and device module17, referring to FIG. 1B, in various embodiments, the control apparatus10 may be comprised of a single unit having an electronic controller 28that collects data from the sensors 18, completes algorithms totranslate the data into a desired movement commands, and sets and runsthe plurality of prosthetic actuators 13 to achieve the desired movementof the prosthetic device 12.

Referring to FIG. 2, wherein like numerals represent like elements, oneembodiment of the control apparatus 10 includes a foot sensor module1015. In some embodiments, the foot sensor module 1015 comprises one ormore inner sole sensors 1018, the sensor CPU 1019 and the sensorcommunicator 1020. In this embodiment, at least one inner sole sensor1018 is positioned in a housing 1032 of a joystick 1034 and sensesmotion of the joystick 1034, which has at least two degrees of freedom.The joystick 1034 is placed on a sole 1036 of footwear 1038, andconnected to the sensor CPU 1019.

Referring to FIGS. 3 and 4, in some embodiments, the joystick 1034 ispositioned between a big toe 1040 and an index toe 1042 of the foot1021. Referring to FIGS. 5-6, the joystick 1034 has a rod 1044 centeredthrough and operatively connected to the housing 1032 such that rod 1044has two degrees of freedom. The sensor 1018, as shown in FIG. 6, ispositioned inside the housing 1032 and below rod 1044. While thedimensions of housing 1032 may vary, in the exemplary embodiment, it hasdimensions small enough to fit comfortably between the user's big toe1040 and index toe 1042 and small enough to fit inside footwear 1038.Housing 1032 may also include and/or have mounts 1046 so that joystick1034 may be attached to the sole 1036 of footwear 1038. The dimensionsof rod 1044 may vary, but in the exemplary embodiment, the rod 1044 maybe long enough for the user to grasp it between the big toe 1040 andindex toe 1042. In the exemplary embodiment, the joystick 1034 may bethick enough that when the user presses against it, the joystick 1034will not break. Rod 1044 may be made of stainless steel or other durablematerial. A magnet 1048 may be placed at the end of rod 1044 disposedinside the housing 1032. The sensor 1018 may be connected to the sensorCPU 1019, shown in FIG. 2, which generates user input signals from thesensor data 1018. The sensor module communicator 1020 of the foot sensormodule 1015 then transmits the user input signals to the device module17, shown in FIG. 1A, through wired connections or wirelessly, forexample, but not limited to, through Bluetooth®, RF communication, orother similar wireless connections. Sensor 1018 detects the position ofrod 1044 and relays that information to the sensor CPU 1019, shown inFIG. 2. Sensor 1018 may be a cross-axial sensor or other similar sensor.The foot sensor module 1015 may impart wireless control to theprosthetic device 12, shown in FIG. 1A.

In the embodiment shown in FIGS. 2-6, the user grips rod 1044 with thebig toe 1040 and index toe 1042 and presses against the rod 1044 tocontrol a direction of movement of the prosthetic device 12, shown inFIG. 1A, or another associated device, such as movement of a mouse on acomputer screen, movement of a car, or movement of other similarremote-controlled devices. The user may also move rod 1044 by placingthe big toe 1040 on top of rod 1044 and pressing the rod 1044 in thedesired direction. As the user moves rod 1044, sensor 1018 detectsdisplacement of the magnet 1048 at the end of rod 1044, and thus detectsthe direction the user is moving rod 1044. That displacement informationis then relayed to the sensor CPU 1019, which translates the movement ofrod 1044 into a desired movement of the associated device. The sensormodule communicator 1020 then communicates the displacement informationto the device module 20, shown in FIG. 1A, which commands movement ofthe associated device. The foot sensor module 1015 has control of twodegrees of freedom such as left and right, up and down, or forward andbackward. The foot sensor module 1015 may also be used as a discreteswitch, such as an on/off switch control a mode of the associateddevice, as will be discussed in greater detail below.

Referring to FIG. 7, another embodiment of the foot sensor module 2025may include a ball joystick 2034. The ball joystick 2034 includes aroller ball 2044 instead of the rod 1044. In this embodiment, the usermay control the prosthetic device 12, shown in FIG. 1A, by moving thebig toe 1040 across the roller ball 2044. For example, if the balljoystick 2034 is programmed to control left and right movement of aprosthetic arm, when the user presses the left side of roller ball 2044,the prosthetic arm will move to the left. Similarly, when the userpresses the right side of roller ball 2044, the prosthetic arm will moveto the right.

Referring to FIGS. 8A, 8B and 9, another embodiment of the foot sensormodule 3015 is shown. In this embodiment, foot sensor module 3015includes an inner sole 3036 having sole sensors 3018, positioned atvarious points on the inner sole 3036. The sole sensors 3018 may be ofthe type such as pressure sensors, force sensors, or the like. Thesensors 3018 are affixed to an underside 3050 of the inner sole 3036.The device module 17, shown in FIG. 1A, may be programmed to controlvarious functions of the prosthetic device 12, shown in FIG. 1A, basedon the input from each sole sensor 3018. Although shown with multiplesole sensors 3018, as few as one sole sensor 3018 may be used, in whichcase the lone sole sensor 3018 may function as a discrete on/off switch(and in some embodiments where multiple sensors are used, one or moresensors 3018 may function as on/off switches). Those skilled in the artwill appreciate that by adding more sole sensors 3018 to inner sole3036, the difficulty in independently controlling the movement of andpressure applied to each sensor 3018 must be taken into consideration.Using two sole sensors 3018, the control apparatus 10, shown in FIG. 1A,will have two degrees of freedom, either up and down, left and right,forward and backward, open and close or other similar discrete function.Using four sole sensors 3018, the control apparatus 10, shown in FIG.1A, will have four degrees of freedom with the ability to move forward,backward, left, and right or up, down, left, and right. Using six solesensors 3018, the control apparatus 10, shown in FIG. 1A, will have 6degrees of freedom with the ability to move up, down, left, right,forward, and backward. In various embodiments, one or more of thesesensors 3018 may also function as discrete switches, for example, toallow the user to select various hand grips, as will be discussed ingreater detail below.

In the exemplary embodiment shown in FIGS. 8A, 8B and 9, foot sensormodule 3015 has four sole sensors 3018 placed on the underside 3050 ofthe inner sole 3036. FIG. 8B shows where the sole sensors 3018 are inrelation to a user's foot 3021: one under the big toe 3040, one underthe left side of the foot 3021, one under the right side of the foot3021, and one under the heel of the foot 3021. The sole sensor 3018under the big toe 3040 may control movement of the arm forward, the solesensor 3018 under the left side of the foot 3021 may control movement ofthe arm to the left, the sole sensor 3018 on the right side of the foot3021 may control movement of the arm to the right, and the sole sensor3018 under the heel may control movement of the arm backward.

In alternative embodiments, the sole sensors 3018 could be placed underother parts of the foot 3021. For example, referring to FIGS. 10A and10B, the underside 3050 of the inner sole 3036 might have one solesensor 3018 under the ball of the foot 3021 and three sole sensors 3018under the heel of the foot 3021.

Regardless of the sensor placement, in operation, the embodiments shownin FIGS. 8A-10 operate in a similar fashion. The sensor CPU 3019receives input data from the sole sensors 3018 and filters and/orconverts the data to generate user input signals. The user input signalsare transmitted to the device module 17, shown in FIG. 1A, by the sensormodule communicator 3020. The device CPU 26, shown in FIG. 1A, thencalculates prosthetic device actuator commands based, at least in part,on the user input signals from the sensor module 3015 and commands theprosthetic controller 27, shown in FIG. 1A, to control the associateddevice, such as a mouse on a computer screen, a robot, or a prostheticlimb in accordance with the device actuator commands. Wires 3052, shownin FIG. 8A, may connect the sensors 3018 to the sensor CPU 3019, whichmay be attached to the shoe. The sensor module 3015 may be connected tothe device module 17, shown in FIG. 1, by wires or wirelessly, forexample, through a blue tooth device or other wireless communicationsystem.

In operation, as the user presses down on the sole sensors 3018, apressure or force pattern of the foot 3021 is created, depending on thesole sensor placement. The sensor module 3015 measures the change inpressure applied by the user, and relays the pattern to the devicemodule 17, shown in FIG. 1. The device module 17, shown in FIG. 1A,translates the pattern into the prosthetic actuator command. Forexample, the device module 17, shown in FIG. 1A, may command movement ofthe associated device in the form of a velocity change or a positionchange using an equation, such as ΔP={right arrow over(V)}_(to be changed) for velocity change or ΔP=X_(to be changed) forposition. For example, with the foot sensor module 3015 of theembodiment of FIGS. 8A and 8B, if the user desires to move theprosthetic arm up, he might press down on the sole sensor 3018 that isbelow the big toe 3040. This creates a pressure pattern that is thenrelayed to the device module 17, shown in FIG. 1A, and translated intoan upward movement of the prosthetic arm. If the user desires to movethe prosthetic arm down, he might press down on the sole sensor 3018under the heel, which creates a different pressure pattern that isrelayed to the device module 17, shown in FIG. 1A, and translated into adownward movement of the prosthetic arm.

Although described for exemplary purposes as providing directionalcontrol, sole sensors 3018 may also provide proportional control. Forexample, with sole sensors 3018 that are pressure sensors or forcesensors, the amount of pressure or force exerted on them may betranslated into a speed at which the controlled device moves. Referringto FIGS. 8A, 8B and 9, for the foot sensor module 3015, if the userdesires to move the prosthetic arm quickly across the body from left toright, he might heavily press sole sensor 3018 on the right side ofinner sole 3036. Alternatively, if the user desires to move theprosthetic arm slowly across the body from left to right, he mightlightly press sole sensor 3018 on the right side of inner sole 3036.Accordingly, the device actuator commands generated by the device module17, shown in FIG. 1A, may vary depending on the magnitude of thepressure or force applied by the user to the sole sensors 3018, which isdissimilar to sensors that act only as switches, i.e., where no materhow hard the sensor is pressed, the output movement does not change.

With pressure sensors or force sensors, the user has better kinematiccontrol of the prosthesis for smoother, less jerky, movements. The useris not limited to two movements of strictly up and down or left andright, but is rather able to control both the speed and direction of themovement. Additionally, the user may engage multiple sole sensors 3018simultaneously to give a combined motion (e.g. up and left). Forexample, in the embodiment shown in FIGS. 10A and 10B, the foot sensormodule 3015 has three sole sensors 3018 under the heel that control theleft, right, and backward movement of the prosthetic device 12, shown inFIG. 1A. As the user rolls the heel across the sole sensors 3018 fromright to left, the prosthetic device 12, shown in FIG. 1A, will movesmoothly in a similar sweeping backward movement. Without these solesensors 3018, the prosthetic device 12, shown in FIG. 1A, would firsthave to move from left to right, stop, and then move backward, resultingin a choppy motion.

Referring to FIGS. 11A-13, in an alternative embodiment of the footsensor module 3015, the foot sensor module 3015 may additionally havetop sensors 3054 placed on a topside 3056 of the sole 3036. Thisembodiment may have sole sensors 3018 on the underside 3050 of innersole 3036 as well as the top sensors 3054 on the topside 3056 of innersole 3036. In such an embodiment, top sensors 3054 may act as discreteor proportional switches and may be placed under toes or other parts ofthe foot 3021 that will not significantly affect the pressure or forcereadings of sensors 3018 on the underside 3050 of inner sole 3036. Forexample and still referring to FIGS. 11A and 11B, when used to control aprosthetic arm, top sensors 3054 act as mode switches, located on thetopside 3056 of inner sole 3036 under the index toe 3042 and little toe3058. The top sensor 3054 under the index toe 3042 may be pressed tosignal the device module 17, shown in FIG. 1A, that the foot sensormodule 3015 is in an arm mode or a bulk mode and will be moving certainactuators of the prosthetic arm, such as shoulder, humerus and/or elbowactuators, while locking other actuators in position such as finger andwrist actuators. The top sensor 3054 under the little toe 3058 may thenbe pressed to switch to a hand grasping mode, which signals the devicemodule 17, shown in FIG. 1A, that the foot sensor module 3015 is beingused to change the type of hand grasp by controlling finger and/or wristactuators, while locking one or more of the bulk movement actuators inposition. In other applications, such as using the foot sensor module3015 to drive a cursor on a computer screen, these top sensors 3054might be used to signal as left and right mouse buttons.

Referring to FIGS. 14 and 15, another alternative embodiment of the footsensor module 3015 utilizing sole sensors 3018 may additionally use shoesensors 3060, which may be placed above the toes on an inner portion ofa roof 3037 of footwear 3038. In such an embodiment, shoe sensors 3060may act as discrete switches. For example, in addition to sole sensors3018 on the underside 3050 of sole 3036, the foot sensor module 3015 mayhave the top sensor 3054 on the top surface of sole 3036 below the bigtoe 3040 and shoe sensors 3060 on the inner surface of the roof of theshoe 3038 above the big toe 3040 and index toe 3042. The top sensor 3054and shoe sensors 3060 may be programmed to switch modes. For example,pressing the big toe 3040 up against the shoe sensor 3060 may set thedevice module 17, shown in FIG. 1A, to arm bulk mode or gross mode,wherein the foot sensor module 3015 may be used to control the bulkmovement of the prosthetic arm as will be discussed in greater detailbelow. Alternatively, pressing the big toe 3040 down against the topsensor 3054 may set the device module 17, shown in FIG. 1A, to a wristmode to control only the wrist of the prosthetic arm or to a finessemode in which only the wrist and hand actuators are controlled. Once inthe desired mode, the sole sensors 3018 could then be used to controldesired movements of the prosthetic device 12, shown in FIG. 1A. Theshoe sensors 3060 may also be used to control other features of aprosthetic, such as opening/closing a hand or acting as an on/offswitch. Thus, a body input signal transmitted from a particular sensorof the foot sensor module 3015 could be used by the device module 17,shown in FIG. 1A, to command a variety of movements of the prostheticdevice 12, shown in FIG. 1A, depending upon the selected mode.

Although the foot sensor module 3015 has been shown and described withrespect to the detailed embodiments thereof, it will be understood bythose skilled in the art that various changes in form and detail thereofmay be made without departing from the spirit and scope of theinvention. For example, the sensors may be attached to the inner liningof a sock or may be directly attached to a shoe.

Referring to FIGS. 16 and 17, in another embodiment, the sensor module15 of the control apparatus 10 may include a residuum joystick 4034,having a frame 4062 and residuum sensors 4018. Referring to FIG. 18, inthis embodiment, the residuum joystick 4034 may be attached to aprosthetic support 4064 so that a user's residuum (not shown) may extendinto the residuum joystick 4034. The user may then control theprosthetic device 12, shown in FIG. 1A, by moving the residuum (notshown) to activate the residuum sensors 4018.

In this embodiment, as shown with four residuum sensors 4018 (althoughin various other embodiments, greater than four or less than foursensors may be used), the user may control the movement of theprosthetic device 12, shown in FIG. 1A, in two degrees of freedom, suchas vertical movement and horizontal movement. Referring to FIG. 19, aresiduum 4066 extends into the residuum joystick 4034 having residuumsensors 4018. As shown, the residuum 4066 is not in contact with theresiduum sensors 4018, so the prosthetic device 12, shown in FIG. 1A,will remain stationary. As shown in FIG. 20, the user may generate bodyinput signals transmitted from the sensor module 15, shown in FIG. 1A,to the device module 17, shown in FIG. 1A, by, for example, moving theresiduum 4066 to engage the right residuum sensor 4018. The signalgenerated by engaging the right residuum sensor 4018 may be used by thedevice module 17, shown in FIG. 1A, to command the prosthetic device 12,shown in FIG. 1A, for example, the device module 17 may command theprosthetic device 12 to move to the right. Similarly, as shown in FIG.21, the user may move the residuum 4066 forward and to the left,engaging two residuum sensors 4018 to signal the device module 17, shownin FIG. 1A, to move the prosthetic device 12, shown in FIG. 1A, up andto the left.

The residuum sensors 4018 may alternatively be used as discreteswitches. For example, one residuum sensor may be used to switch betweena bulk mode in which bulk movement of the prosthetic arm is controlledand a finesse mode in which only hand and wrist actuation of theprosthetic arm is controlled.

The residuum input may provide physical feedback to a user. Thus, addingto spatial and other types of feedback a user may experience. Thus, theresiduum input may enhance the control by the user. The residuum inputmay also be used for proportional and/or position control.

Another embodiment of the control apparatus 10, shown in FIG. 1A, useskinematic mapping, sensing head and body movement, to control theprosthetic device 12. The user moves the head and body in coordinationto select a point in space where they desire the prosthetic device 12 tomove. Head movement is slow, intentional and decoupled from a majorfunction, which makes it ideal for prosthetic control. This kinematicmapping control may be a mode that may be selected by the user by, forexample, but not limited to, a double click of a switch.

Referring to FIG. 22, a kinematic mapping sensor module 5015 featuresthree body sensors 5018 in three locations, the shoulder 5025, the head5022, and the waist 5024. In this way, two body sensors 5018 are on thebody of the user and the other body sensor 5018 is on the head 5022. Forexample, a hat 5068 may hold one body sensor 5018 or, alternatively, thehead body sensor 5018 may be mounted above an ear as a separate wirelessunit. One body sensor 55018 may be incorporated into a belt 5070 or apack (not shown) strapped onto the midsection of the user and anotherbody sensor 5018 may be included on a prosthetic support 5071 mounted tothe user's shoulder 5025.

This embodiment uses inertial sensors as body sensors 5018. These threebody sensors 5018 may be used to detect up to six multiple degrees offreedom. Specifically, the body sensors 5018 may detect head yaw 5072,head roll 5074, head pitch 5076, torso yaw 5078, torso roll 5080 andtorso pitch 5082. Additionally, theses body sensors may detect x, y, andz plane positioning. These sensors may also act as velocity accelerationgyros, accelerometers, angular velocity and magnetometers. Althoughshown as inertial sensors, the body sensors 5018 may also be shapesensors that may detect body flex.

Still referring to FIG. 22, in this embodiment of the sensor module5015, the control apparatus 10, shown in FIG. 1A, assumes a fixed pointof rotation at the middle of the prosthetic hand and creates a referencesphere around the fixed point. User preference determines the locationof the fixed point by allowing the user to zero out the system withspecific body input sensed by body sensors 5018, such as looking around.Then the user looks at a point, about which the sphere is created. Bychoosing where the fixed point of rotation is, the user customizes andorients the movement path. To select the fixed point and sphere, head5022 rotation specifies an angle and body lean or shoulder 5025 rotationspecifies radius.

Although the various embodiments of sensor modules 15 have beendescribed separately herein for simplicity, it should be understood bythose skilled in the art that the various embodiments may be used incombination to achieve the desired prosthetic control. For example, thefoot sensor module 3015, shown in FIG. 9, may be used in conjunctionwith another sensor module (and/or control system), such as an inertialsensor module (and/or control system), a shoulder joystick, and/or anEMG sensor module (and/or control system).

Referring back to FIG. 1A, as discussed above, the device module 17 mayuse the body input signals from the sensor module 15 to control theprosthetic device 12 in a variety of different control modes, selectableby the user, to achieve different functionalities from the prostheticdevice 12. For example, a control method of the prosthetic device 12 mayinclude a bulk mode and a finesse mode. Bulk mode includes movement ofthe prosthetic device 12 into the general vicinity desired by the user,for example, by moving the palm of the prosthetic hand to a desiredpoint in space. Thus, for example, bulk mode for a prosthetic arm mayinclude actuation of the shoulder, humerus and/or elbow actuators and/orwrist. The terms bulk movement, gross movement and gross mode as usedherein are synonymous with the term bulk mode.

Finesse mode in this embodiment relates to the ability to manipulate anobject (not shown) and specifically relates to operating a prosthetichand and a prosthetic wrist. Finesse mode may be used to achieve wristrotation, inflection, deviation and hand gripping. Thus, the finessemode allows the prosthetic hand to grasp or grip the object. A grasp orgrip refers to an orientation of the prosthetic hand's overall handpattern, as will be discussed in greater detail below. The grip may beactivated by the user to hold and manipulate the object. The termsfinesse movement, fine movement and fine mode as used herein aresynonymous with the term finesse mode.

The current method uses bulk movement to allow the user to position theprosthetic arm at a specific point in a three-dimensional space (x, y,and z components). Once the prosthetic arm has reached the desiredlocation, i.e. the specific point, finesse movement allows the user tomanipulate the prosthetic hand and grip the object.

Both bulk and finesse movements are determined using the various controlapparatuses described herein. The user determines a point that they wantthe prosthetic arm to reach and relative to a control input, theprosthetic arm moves to that point in space. This type of bulk movementresults in simultaneous degree of freedom movement.

For example, in an embodiment with head control, the head moves andcontrols one joint of the prosthetic arm, resulting in one action. Theinput is head movement; the output is movement of the prosthetic arm.Similarly, referring back to FIG. 15, in an embodiment having the footsensor module 3015, the user may apply pressure with different parts ofthe foot 3021 to sensors 3018, to control the bulk movement of theprosthetic arm. The user may then engage the shoe sensor 3060 to switchfrom bulk movement to finesse movement, and then use sensors 3018 tocontrol the finesse movement of the prosthetic arm. This method allowsthe user to alternate between bulk movement and finesse movement.

In one embodiment, the device module 17 commands shoulder deflection andextension, elbow flexion and extension, and humerus rotation of theprosthetic arm in bulk mode. Additionally, depending on the severity ofthe amputation, shoulder abduction and adduction may also be controlledin bulk mode. In finesse mode, the device module 17 may command wristrotation flexion, deviation and extension, and hand manipulation,including thumb and finger movement. In finesse mode, pressure and forcesensors measure the distribution of weight and may be used to detectinput specific to the grasp. The distribution of weight on the footsensors may deliver specific input allowing the device module 17 toselect the desired grip. Alternatively, in another embodiment, headposition may be used to select the grip.

Although described with regard to a shoulder disarticulation amputee, itshould be understood by those skilled in the art that the controlsystems and methods described herein may be adapted to be used for anyprosthetic strapped onto the body. For example, for an elbow jointdisarticulation amputee (direct control of just elbow joint) finessecontrol may be used for wrist and hand manipulation.

In some embodiments, the control apparatus 10 may include other modes,such as: an off mode, a home mode, and a hold mode. Any of the varioussensors described herein may be programmed for mode selection.

In the home mode, the prosthetic device 12 is in a preset location, suchas by the side of the user and the device module 17 is not sending anycommands to the prosthetic device 12. Thus, the prosthetic device 12 isnot moving. In the hold mode, the prosthetic device 12 maintains a fixedposition. The term standby mode has also been used herein and issynonymous with hold mode. In one embodiment, the hold position appearsas though it is not receiving any input, but rather, the last positiondata is continuously sent to the prosthetic device 12 to activelymaintain the position.

In an alternative embodiment of the hold mode, a hold command may besent that engages various brakes within the prosthetic device 12, ratherthan continually sending the same coordinates, freeing the system to doother functions instead of continuously calculating the last position.This improves the control apparatus 10 by conserving power.

Referring to FIG. 23, one embodiment of the control method of thecontrol apparatus 10 includes operating the prosthetic device 12 in homemode S1, then in bulk mode S2, then in finesse mode S3, and then in bulkmode S4. This allows the user to enter bulk mode and move the prostheticarm to the desired location, then enter finesse mode to move theprosthetic hand and wrist to manipulate the object as desired, and thenreturn the arm to home mode.

Referring to FIG. 24, an additional embodiment may include operating thecontrol apparatus 10 in home mode S5, then in bulk mode S6, then infinesse mode S7, then in hold mode S8, and then in bulk mode S9. Thisallows a user to move the prosthetic arm to the desired location andmanipulate the object, then the user is able to hold the object in thedesired position before the prosthetic arm is returned to home mode.

Referring to FIG. 25, in these embodiments having sensors 18, a personusing the control apparatus puts the prosthetic arm on and simple setupstate procedure is executed to quickly calibrate the prosthetic arm. Forinstance, orientation sensors in the prosthetic arm may provide positioninformation to the device module 17 to identify the starting position ofthe prosthetic arm S10. The device module 17 then tares the sensors 18to zero them out, so that their rotations are in respect to their tarredposition S11. The body sensors are then read to get the user's perceivedZ and Y axis S12. A calibration step is then run where the Z axis isprojected on the normal plane with the Y axis to get the X axis S13. Thebody sensors are then read again to identify the coordinates for thehome mode S14. Then the control apparatus 10 is ready to be operated.

Referring to FIG. 26, when the control apparatus 10 is in a deactivatedstate such as in home mode or hold mode, prior to enabling movement, thedevice module 17 may use transformation sensors in the prosthetic armtare the body sensors to zero them out, so that their rotations are inrespect to their tarred position S15. The body sensors are then read toget the user's perceived Z and Y axis, and the Z axis is projected onthe normal plane with the Y axis to get the X axis S16. Once theperceived axes are known, the sensors 18 are activated and may be usedin bulk mode and/or finesse mode. The transformation sensors use thefixed point of the cylindrical mapping system and the lengths of eachprosthetic arm component to determine when the arm has achieved thedesired point in space. In various other embodiments, a sphericalmapping system and/or a Cartesian coordinate system may also be used.

Referring to FIG. 27, a control method for embodiments using kinematicmapping, such as that shown in FIG. 22, is shown. When the sensors havebeen activated, the sensors identify the desired coordinates for theprosthetic arm to move to S17. Once the fixed point is specified, thedevice module 17 goes through equation calculations (which in someembodiments, may include quadratic equation calculations) to calculatethe best velocity and direction vector for getting the target sphereand/or coordinate and/or point in three-dimensional space to line upcorrectly S18. The device module 17 then goes through dot products todetermine the necessary angles for the shoulder, elbow and humeralprosthetic movement S19. Based on those calculated angles, the arm ismoved to reach the target sphere S20. Once the sensors determine thatthe target sphere has been reached, the arm movement is stopped S21.

In an alternative embodiment utilizing kinematic mapping, there is aclick and go mode. Thus, in the click and go mode, if the user wants tomove the prosthetic device 12 to an object, they may look at a point inspace where they want the prosthetic device 12 to go, and then engage asensor, such as residuum sensor 4018 shown in FIG. 19, that activatesthe click and go mode. Once activated, the body sensors determine wherethe head was looking and where the body leaned, and coordinates are sentdirecting the prosthetic device 12 to go to that location. Click and gomode may use the same sensor set for controlling bulk movement asfinesse movement. For instance, once the bulk movement begins, the headsensor 5018 may then control the finesse movement.

In another embodiment, by using accelerometers and body sensors 5018,the control apparatus 10, shown in FIG. 1A, is able to identify thecenter of gravity in relation to the body sensor 5018 on the shoulder.From that, the device module 17, shown in FIG. 1A, sending anglecommands to the prosthetic arm knows where the end of prosthetic arm isand knows where the gravity vector with respect to the end of the armis. Therefore, the device module 17, shown in FIG. 1A, may rotate thewrist with respect to the gravity vector to maintain an object (notshown) within the prosthetic hand in an upright position.

In an alternate embodiment using body sensors, the user could put thesensor on only their head, using the sensor to three-dimensionally mapthe desired movements. This would decrease the number of sensorsrequired to control the prosthetic.

The control apparatus 10 may control sensitivity of movement in that thedevice module 17 may vary the degree that sensor input is translated tomovement output. Additionally, The sensor input may be required to meeta threshold value before movement output is sent to the prosthetic.

In some embodiments, there may also be an arm swing mode, allowing theprosthetic arm to move in harmony with the body while walking. When theuser is going to use the arm, it is in the home/off position, and swingmode may be activated by engaging a sensor 18 or by detecting a specificmotion or orientation with the body sensor 5018. In some embodiments,swing mode may also be activated by a manual switch and/or leverdisengaging mechanical engagement of a portion of the prosthetic device12.

Switching modes or selecting commands may be accomplished by engagingsensors 18 acting as discrete switches, by specific body motion such asticks or head movement, by standard EMG signals using EMG electrodes, byshoulder or back movements, or by any other similar switching mechanismthat may be programmed into the control apparatus 10 including, but notlimited to, macros and/or other commands beyond direct kinematicmovement or mode.

The sensors 18 may be disposed in various locations for detecting bodyinput 16 to control the movement of the prosthetic device 12, such as infootwear. The control apparatus 10 may utilize wireless communicationbetween the sensors 18, the sensor module 15, the device module 17 andthe prosthetic device 12, simplifying the control apparatus 10 for theuser. The sensors 18 may act as discrete switches to control operationalmodes of the prosthetic device and/or the control apparatus 10 may movethe prosthetic device 12 proportionally to the body input 16 sensed bythe sensors 18. The sensors 18 may also be disposed in a prostheticsupport apparatus 4064, allowing user to provide body input 16 to thesensors 18 with the residuum 4066.

Each sensor 18 may sense a variety of body inputs 16 such as pressureand rate of pressure change. Therefore, body input 16 from one sensor 18may be translated by the device module 17 into multiple forms ofmovement information, such as direction and speed of movement.

Referring to FIG. 28, in various embodiments, rather than a series ofdiscrete sensors 3018 as shown in FIGS. 8A-15, a sensor grid 6084 may beincluded in footwear 6038 to generate a pressure profile for the user'sfoot 6021. The sensed pressure profile may be sent by the sensor CPU6019 to the device module 17, shown in FIG. 1A, and used by the devicemodule 17, shown in FIG. 1A, to command the prosthetic device 12, shownin FIG. 1A. Thus, by changing the pressure profile of the foot 6021, theuser may be able to command different functions from the prostheticdevice 12, shown in FIG. 1A. Although shown on the underside 6050 of theinner sole 6036, the sensor grid 6084 could also be include on thetopside 6056 of the inner sole 6036 or could be integral with the innersole 6036.

Referring to FIG. 29, the sensor grid 6084 includes a plurality of zones6086 in which pressure may be detected, to generate the pressure profilefor the user's foot 6021, shown in FIG. 28. For example, force sensingresistors may be used to form the plurality of zones 6086, therebyallowing the pressure on each zone to be detected separately. Inaddition to force sensing resistors, in other embodiments, otherpressure or force sensors may be used to form the plurality of zones6086, for example, strain gauges, air bladders and any other similarforce sensors.

Referring to FIG. 30, the pressure profile may show that region 6088, onone side of the user's footwear 6038, has higher pressure relative tothat of region 6090, on the opposite side of the user's footwear 6038.This pressure profile may be used to command a particular motion fromthe prosthetic device 12, shown in FIG. 1A, for example, the pressureprofile may move the prosthetic device 12, shown in FIG. 1A, to theright. Similarly, a pressure profile with a higher relative pressure inregion 6090 to that of region 6088 may be used to command a differentmotion from the prosthetic device 12, shown in FIG. 1A, for example,movement of the prosthetic device 12, shown in FIG. 1A, to the left. Thepressure profile may also show that region 6092, at the front of theuser's footwear 6038, is greater or lower than the pressure of region6094, at the rear of the user's footwear 6038. Thus, the pressureprofile of the front relative to the rear of the user's footwear 6038may be used by the device module 17, shown in FIG. 1A, to provide theprosthetic device 12, shown in FIG. 1A, with two additional degrees offreedom, such as forward and rearward motion. Various pressure profilesdetected by the sensor grid 6084 may also be used as switches; forexample, a pressure profile with a high pressure in region 6090 relativeto region 6088 may be used to change between modes, such as bulk modeand finesse mode. Similarly, different pressure profiles may be used toselect hand grips or to scroll through a list as will be discussed ingreater detail below.

Thus, referring back to FIG. 28, the sensor grid 6084 provides thecontroller apparatus 10, shown in FIG. 1A, with the ability to controlmovement of the prosthetic device 12, shown in FIG. 1A, in at least twodegrees of freedom, to control multiple switches or to control somecombination of movement and switching. This embodiment may be moredesirable than the embodiments with multiple sensors 3018, shown inFIGS. 8A-15, since in the multi-sensor approach the user's foot 6021 maymove around in the footwear 3038, shown in FIG. 15, making it moredifficult for the user to locate and activate the sensors 3018. Thesensor grid 6084 overcomes the issue of sensor location and activation,by using variations in the pressure pattern formed by the user's foot6021 to command the prosthetic device 12, shown in FIG. 1A. Thus, theuser must only shift weight in a desired direction to change thepressure profile, rather than locating discrete sensors within thefootwear 6038. Additionally, using pressure pattern recognition, thedevice module 17, shown in FIG. 1A, may determine the direction theuser's weight is being shifted, thereby allowing small movements to bedetected so that the user is not required to exaggerate their movements;rather, the sensor grid 6084 will sense small or micro movements andcommand the prosthetic device 12, shown in FIG. 1A, accordingly.Additionally, in various embodiments, the sensor grid 6084 may beimplemented along with shoe sensors 6060 for the use as discreteswitches as discussed above.

Referring to FIG. 31A, the sensor module 7015 of the control system 7010may include one or more Inertial Measurement Units (IMUs) 7096 in placeof, or in addition to, the one or more sensors 7018. The one or moreIMUs 7096 detect orientation, as will be discussed in greater detailbelow, which may be transmitted to device module 7017 for commanding theassociated prosthetic device 7012. Thus, by altering the orientation ofthe IMU 7096, the user may control the prosthetic device 7012 in adesired manner. Referring to FIG. 31B, in some embodiments wheremultiple IMUs 7096 are attached to different body parts, it may bedesirable to provide separate sensor modules 7015 for each IMU 7096 todecouple to IMUs 7096 from each other. In these embodiments, each sensormodule 7015 may communicate with the device module 7017 and the devicemodule 7017 uses the body input signals provided from each sensor module7015 to command the associated prosthetic device 7012.

Referring to FIG. 32, in some embodiments, the IMU 7096 may determinethe orientation of the user's foot 7021. In some embodiments,particularly where an increased number of control inputs is desired, oneIMU 7096 may be used on each foot 7021 of the user (the term “feet” or“foot” is a general description, in some embodiments, the IMU 7096 maybe placed on a user's ankle or ankles or on the user's leg or legs. Insome embodiments, the IMU(s) 7096 may be placed on any part of a userindicative of the movement of the foot/feet, including, but not limitedto, affixed to the user's clothing or footwear 7036). In someembodiments, IMUs 7096 may be placed at other locations on the userincluding but not limited to the user's arm, head, or the like. Each IMU7096 is a device capable of sensing motion using a combination ofsensors as will be discussed in greater detail below.

Referring to FIG. 33, in some embodiments, the IMUs 7096 may be acommercially available unit such as a MICROSTRAIN® 3DM-GX1® byMicrostrain, Inc., Williston, Vt. In some embodiments, a variety ofother inertial measurement units may be implemented, such as thosedescribed in the U.S. patent application entitled SYSTEM, METHOD ANDAPPARATUS FOR ORIENTATION CONTROL, filed on the same day as the presentapplication and assigned to the same assignee, which is herebyincorporated by reference in its entirety. In these embodiments, the IMU7096 may be accommodated in a housing 7098 having a base 7100 and acover 7102, which interface to enclose the IMU 7096 within a housingcavity 7106. The cover 7102 may be connectable to the base 7100 by aplurality of screws 7109 or other known fastening means, such as a snapfit, one or more latches, or the like. In the exemplary embodiment thehousing 7098 may measure approximately 0.61 in.×0.65 in.×0.22 in. Thebase 7100 of the housing 7098 may include an electronics orifice 7110,through which the IMU 7096 within the housing 7098 may be connected tothe sensor CPU 7019 and the sensor module communicator 7020, shown inFIGS. 31A and 31B, for example, through one or more wires 7112. The oneor more wires 7112 may also connect the IMU 7096 to a battery (notshown) for powering the IMU 7096.

Referring to FIG. 34, the IMU 7096 may include one or moreaccelerometers 7114 and/or one or more gyroscopes 7116, to measureorientation of the IMU 7096 relative to a gravitational direction G,shown in FIG. 32, including, but not limited to, sensing type, rate, anddirection of the orientation change of the IMU 7096. The IMU 7096 has anoutput 7118 to facilitate connection of the IMU 7096 to the sensor CPU7019, sensor module communicator 7020 and battery (not shown) throughthe electronics orifice 7110, shown in FIG. 33. The one or moreaccelerometers 7114 and the one or more gyroscopes 7116 may beelectrically connected to the output 7118 by one or more circuit boards7120. As discussed above, the sensor module communicator 7020 mayinclude a radio or Bluetooth® transmitter for wirelessly transmittingsignals to the device module 7017, shown in FIGS. 31A and 31B, or thesensor module communicator 7020 may be hardwired to the device module7017, shown in FIGS. 31A and 31B.

Referring back to FIG. 32, the data collected from the at least one IMU7096 may be used by the device module 7017, shown in FIGS. 31A and 31B,in an algorithm to translate orientation of the foot 7021 and/or changesin orientation to a commanded movement of the prosthetic device 7012,shown in FIGS. 31A and 31B. In some embodiments, IMU 7096 may include atleast two accelerometers 7114 detecting acceleration about two axes andat least one gyroscope 7116 for detecting orientation changes about athird axis. Thus, the IMU 7096, in some embodiments, may detectorientation changes about at least three axes, thereby allowing the userto control the prosthetic device 7012, shown in FIGS. 31A and 31B, in atleast three degrees of freedom.

The accelerometers 7114 of each of the IMUs 7096 may be arranged todetect pitch θ_(Pitch) about the X axis relative to the gravitationaldirection G and roll θ_(Roll) about the Y axis relative to thegravitational direction G. The gyroscope 7116, shown in FIG. 34, of eachof the IMUs 7096 is, in some embodiments, arranged to detect yaw {dotover (θ)}_(Yaw) about the Z axis. Thus, by using two IMUs 7096, one IMU7096 on each foot 7021, the user is able to control the prostheticdevice 7012, shown in FIGS. 31A and 31B, in at least six degrees offreedom.

Each IMU 7096 is arranged with one accelerometer 7114 in the Y directionand the other accelerometer 7114 in the X direction. When the IMU 7096is flat, i.e. the Z axis is coincident with the gravitational directionG, gravity, which is an acceleration of 1G in the gravitationaldirection G, only includes a component projected on the Z axis. As theIMU 7096 tilts, a component of gravity is projected onto the X axisand/or Y axis. This tilt is detectable by the accelerometer 7114arranged on the axis upon which the component of gravity is projected.Since 1G is a known value, the arcsin of the value detected by eachaccelerometer 7114 of the IMU 7096 is a proportion of 1G andrepresentative of the pitch θ_(Pitch) and/or roll θ_(Roll).

Although shown in FIG. 32 with the Z axis being coincident with thegravitational direction G, as seen in FIGS. 35 and 36, the Z axis ofeach of the IMUs 7096 may be offset from the gravitational direction G;for example, if the IMU 7096 is not initially situated flatly on theusers foot 7021, if the IMU 7096 shifts during use, or if the user isstanding on an incline, decline or the like. Therefore, the sensormodule 7015 of the present invention may zero the IMUs 7096, as will bediscussed in greater detail below, by setting a pitch offset, θ_(Offset)_(_) _(Pitch), and a roll offset, θ_(Offset) _(_) _(Roll), wheninitialized or reinitialized during use.

Referring to FIG. 37, the pitch θ_(Pitch) detected by the IMU 7096 maybe configured to command the prosthetic device 7012, shown in FIGS. 31Aand 31B. For example, the device module 7017, shown in FIGS. 31A and31B, may command the prosthetic device when:|θ_(Pitch)=θ_(Offset) _(_) _(Pitch)|≧θ_(Threshold) _(_) _(Pitch)where,

θ_(Pitch) is the pitch detected by the IMU 7096 relative to thegravitational direction G;

θ_(Offset) _(_) _(Pitch) is the preset value calibrating the IMU 7096discussed above; and

θ_(Threshold) _(_) _(Pitch) is a present minimum pitch angle that mustbe exceeded to ensure that the detected pitch θ_(Pitch) is a desiredcommand and not due to unintentional movement of the user's foot 7021,shown in FIG. 32.

In one embodiment, the command generated by the device module 7017,shown in FIGS. 31A and 31B, from the pitch θ_(Pitch) may be a switchthat alternates between an “on state” and an “off state” each time|θ_(Pitch)−θ_(Offset) _(_) _(Pitch)|≧θ_(Threshold) _(_) _(Pitch). Inanother embodiment, pitch θ_(Pitch) may command the controller to togglethrough a list of operational control modes, which will be discussed ingreater detail below. For example, each instance that θ_(Threshold) _(_)_(Pitch) is exceeded, the controller may toggle forward through the listif (θ_(Pitch)−θ_(Offset) _(_) _(Pitch)) is a positive value and maytoggle backward, i.e. in reverse, through the list if(θ_(Pitch)−θ_(Offset) _(_) _(Pitch)) is a negative value.

In one embodiment, the command generated by the device module 7017,shown in FIGS. 31A and 31B, may correspond to a movement, M_(Pitch), ofthe prosthetic device 7012, shown in FIGS. 31A and 31B, if|θ_(Pitch)−θ_(Offset) _(_) _(Pitch)|≧θ_(Threshold) _(_) _(Pitch). Forexample, when |θ_(Pitch)−θ_(Offset) _(_) _(Pitch)|≧θ_(Threshold) _(_)_(Pitch) the device module 7017, shown in FIGS. 31A and 31B, may commandmovement at a preset velocity in a preset direction, e.g. the devicemodule 7017 may command upward movement at the preset velocity if(θ_(Pitch)−θ_(Offset) _(_) _(Pitch)) is a positive value and may commanddownward movement if (θ_(Pitch)−θ_(Offset) _(_) _(Pitch)) is a negativevalue. In another embodiment, the movement may be commanded using theequation:M _(Pitch) =k ₁(θ_(Pitch)−θ_(Offset) _(_) _(Pitch))+k ₂where,

k₁ and k₂ are gains that may be preset based on the type of movementdesired. The movement M_(Pitch) may be set to correspond to a variety ofpossible movements of the prosthetic device 7012, shown in FIGS. 31A and31B. For example, M_(Pitch) may be a distance of deflection in a directdirection or a speed of travel in a direction.

Referring to FIG. 38, the roll θ_(Roll) detected by the IMU 7096 mayalso be configured to command the prosthetic device 7012, shown in FIGS.31A and 31B, in a manner similar to that discussed above for the pitchθ_(Pitch). For example, the device module 7017, shown in FIGS. 31A and31B, may command the prosthetic device when:|θ_(Roll)−θ_(Offset) _(_) _(Roll)|≧θ_(Threshold) _(_) _(Roll)where,

θ_(Roll) is the roll detected by the IMU 7096 relative to thegravitational direction G;

θ_(Offset) _(_) _(Roll) is the preset value calibrating the IMU 7096discussed above; and

θ_(Threshold) _(_) _(Roll) is a present minimum roll angle that must beexceeded to ensure that the detected roll θ_(Roll) is a desired commandand not due to unintentional movement of the user's foot 7021, shown inFIG. 32.

In one embodiment, the command generated by the device module 7017,shown in FIGS. 31A and 31B, from the roll θ_(Roll) may be a switch thatalternates between an “on state” and an “off state” each time|θ_(Roll)−θ_(Offset) _(_) _(Roll)|≧θ_(Threshold) _(_) _(Roll). Inanother embodiment, roll θ_(Roll) may command the device module 7017,shown in FIGS. 31A and 31B, to toggle through a list of operationalmodes, which will be discussed in greater detail below. For example,each instance that θ_(Threshold) _(_) _(Roll) is exceeded, the devicemodule 7017, shown in FIGS. 31A and 31B, may toggle forward through thelist if (θ_(Roll)−θ_(Offset) _(_) _(Roll)) is a positive value and maytoggle backward, i.e. in reverse, through the list if(θ_(Roll)−θ_(Offset) _(_) _(Roll)) is a negative value.

In one embodiment, the command generated by the device module 7017,shown in FIGS. 31A and 31B, may correspond to a movement, M_(Roll), ofthe prosthetic device 7012, shown in FIGS. 31A and 31B, if|θ_(Roll)−θ_(Offset) _(_) _(Roll)|≧θ_(Threshold) _(_) _(Roll). Forexample, when |θ_(Roll)−θ_(offset) _(_) _(Roll)|≧θ_(Threshold) _(_)_(Roll) the device module 7017, shown in FIGS. 31A and 31B, may commandmovement at a preset velocity in a preset direction, e.g. the devicemodule 7017 may command movement to the right at the preset velocity if(θ_(Roll)−θ_(Offset) _(_) _(Roll)) is a positive value and may commandmovement to the left if (θ_(Roll)−θ_(Offset) _(_) _(Roll)) is a negativevalue. In another embodiment, the movement may be commanded using theequation:M _(Roll) =k ₃(θ_(Roll)−θ_(Offset) _(_) _(Roll))+k ₄where,

k₃ and k₄ are gains that may be preset based on the type of movementdesired. The movement M_(Roll) may be set to correspond to a variety ofpossible movements of the prosthetic device 7012, shown in FIGS. 31A and31B. For example, M_(Roll) may be a distance of deflection in a directdirection or a speed of travel in a direction.

Referring to FIG. 39, each gyroscope 7116 is able to detect yaw {dotover (θ)}_(Yaw) as the rate of angular rotation relative to the Z axis.Thus, yaw {dot over (θ)}_(Yaw) about the Z axis is detectable by the IMU7096 when the user's foot 7021 moves about the Z axis. Unlike the pitchθ_(Pitch) and roll θ_(Roll), which are each detected relative to a fixedreference, i.e. the gravitational direction G, the yaw {dot over(θ)}_(Yaw) is detected by the gyroscope 7116 with respect to thereference frame of the gyroscope 7116. Thus, the gyroscope 7116effectively resets its frame of reference after each angular deflectionof the IMU 7096. For example, if after moving from the first position P₁to the second position P₂, the user then moves the IMU 7096 to a thirdposition P₃, the yaw {dot over (θ)}_(Yaw) detected by the IMU 7096 asthe IMU 7096 moves from the second position P₂ to the third position P₃would be relative to the second position P₂. This yaw {dot over(θ)}_(Yaw) detected by the IMU 7096 may be configured to command theprosthetic device 7012, shown in FIGS. 31A and 31B.

For example, the device module 7017, shown in FIGS. 31A and 31B, maycommand the prosthetic device 7012 when:|{dot over (θ)}_(Yaw)|≧{dot over (θ)}_(Threshold) _(_) _(Yaw)where,

{dot over (θ)}_(Yaw) is the yaw detected by the IMU 7096; and

{dot over (θ)}_(Threshold) _(_) _(Yaw) is a present minimum yaw angularrotation that must be exceeded to ensure that the detected yaw {dot over(θ)}_(Yaw) is a desired command and not due to unintentional movement ofthe user's foot 7021, shown in FIG. 32, or drifting of the gyroscope7116.

Advantageously, since the yaw {dot over (θ)}_(Yaw) detected by thegyroscope 7116 about the Z axis is relative to the previous position ofthe IMU 7096, rather than a fixed reference frame like the gravitationaldirection G, shown in FIG. 32, a yaw offset is not necessary, as was thecase with detection of the pitch and roll.

In one embodiment, the command generated by the device module 7017,shown in FIGS. 31A and 31B, from the yaw {dot over (θ)}_(Yaw) may be aswitch that alternates between an “on state” and an “off state” eachtime |{dot over (θ)}_(Yaw)|≧{dot over (θ)}_(Threshold) _(_) _(Yaw). Inanother embodiment, yaw {dot over (θ)}_(Yaw) may command the devicemodule 7017, shown in FIGS. 31A and 31B, to toggle through a list ofoperational modes, which will be discussed in greater detail below. Forexample, each instance that {dot over (θ)}_(Threshold) _(_) _(Yaw) isexceeded, the device module 7017, shown in FIGS. 31A and 31B, may toggleforward through the list if {dot over (θ)}_(Yaw) is a positive value andmay toggle backward, i.e. in reverse, through the list if {dot over(θ)}_(Yaw) is a negative value.

In one embodiment, the command generated by the device module 7017,shown in FIGS. 31A and 31B, may correspond to a movement, M_(Yaw), ofthe prosthetic device 7012, shown in FIGS. 31A and 31B, if |{dot over(θ)}_(Yaw)|≧{dot over (θ)}_(Threshold) _(_) _(Yaw). For example, when|{dot over (θ)}_(Yaw)|≧{dot over (θ)}_(Threshold) _(_) _(Yaw) the devicemodule 7017, shown in FIGS. 31A and 31B, may command movement M_(Yaw) ata preset velocity in a preset direction, e.g. the device module 7017 maycommand movement to the right at the preset velocity if {dot over(θ)}_(Yaw) is a positive value and may command movement to the left if{dot over (θ)}_(Yaw) is a negative value. In this exemplary embodimentfor commanding right and left movement, it may also be desirable to haltright and left movement using the detected yaw {dot over (θ)}_(Yaw). Forexample, if the device module 7017 has commanded movement M_(Yaw) to theright, based on a positive {dot over (θ)}_(Yaw), a subsequently detectednegative {dot over (θ)}_(Yaw) that satisfies the relationship |{dot over(θ)}_(Yaw)|≧{dot over (θ)}_(Threshold) _(_) _(Yaw) may generate acommand to stop moving to the right, rather than a command to move tothe left. From the stopped position, another negative {dot over(θ)}_(Yaw) that satisfies the relationship |{dot over (θ)}_(Yaw)|≧{dotover (θ)}_(Threshold) _(_) _(Yaw) would then command leftward movementor, alternatively, a positive {dot over (θ)}_(Yaw) that satisfies therelationship |{dot over (θ)}_(Yaw)|≧{dot over (θ)}_(Threshold) _(_)_(Yaw) would then command rightward movement. Similarly, if the devicemodule 7017 has commanded movement M_(Yaw) to the left, based on anegative {dot over (θ)}_(Yaw), a subsequently detected positive {dotover (θ)}_(Yaw) that satisfies the relationship |{dot over(θ)}_(Yaw)|≧{dot over (θ)}_(Threshold) _(_) _(Yaw) may generate acommand to stop moving to the left, rather than a command to move to theright. From the stopped position, a negative {dot over (θ)}_(Yaw) thatsatisfies the relationship |{dot over (θ)}_(Yaw)|≧{dot over(θ)}_(Threshold) _(_) _(Yaw) would then command leftward movement or,alternatively, a positive {dot over (θ)}_(Yaw) that satisfies therelationship |{dot over (θ)}_(Yaw)|≧{dot over (θ)}_(Threshold) _(_)_(Yaw) would then command rightward movement.

For exemplary purposes, the pitch θ_(Pitch), roll θ_(Roll) and yaw {dotover (θ)}_(Yaw) have been described as commanding specific movements inconnection with FIGS. 35-39. However, it should be understood that thepitch θ_(Pitch), roll θ_(Roll) and yaw {dot over (θ)}_(Yaw) may beprogrammed within the device module 7017, shown in FIGS. 31A and 31B, tocommand a variety of different movements, and in some embodiments, inresponse to the user's preferences and customization, as will bediscussed in greater detail below.

It should be understood that although the use of at least one IMU 7096for control of a prosthetic device 7012, shown in FIGS. 31A and 31B, isdescribed herein, the at least one IMU 7096 may be used in conjunctionwith any one or more various devices and/sensors 7018 to control theprosthetic device 7012. Thus, in some embodiments, the IMU 7096 may beused in conjunction with the sensors 7018, top sensors 60 and/or sensorgrid 6084 discussed above, as well as with an EMG system and/or with apull switch. For example, in various embodiments of present invention,one or more prosthetic shoulder movements may be controlled by theshoulder sensor 5028, shown in FIG. 22, while side to side movements ofthe prosthetic device 7012 may be controlled by sensors 7018, sensorgrids 6084 or IMUs 7096.

Additionally, although an exemplary embodiment of the IMU 7096 isdescribed herein which may be used in the exemplary embodiment of thesystem, apparatus and method for control of a prosthetic device 7012,shown in FIGS. 31A and 31B, in other embodiments, any device capable ofdetermining orientation may be used for the IMU 7096, as should beunderstood by those skilled in the art. For instance, another type ofsensor 18 may be used in the IMU 7096, such as a 3-axis accelerometer, a3-axis magnetometer and/or tilt bulb(s).

In some embodiments, it may be beneficial to include threeaccelerometers 7114 or a three-axis accelerometer, in the IMU 7096 alongwith at least one gyroscope 7116 for detecting orientation changes aboutat least three axes and for enabling walk detection. In an embodimentwith IMU 7096 having three accelerometers 7114, the IMU 7096 generatesoutput relating to pitch θ_(Pitch), roll θ_(Roll) and yaw {dot over(θ)}_(Yaw) in substantially the same manner discussed above inconnection with the IMU 7096 having two accelerometers 7114. However,with the third accelerometer 7114, the IMU 7096 may provide the controlapparatus 7010, shown in FIGS. 31A and 31B, with walk detectioncapability.

Referring back to FIGS. 31A and 31B, when using the IMU 7096 for controlof the prosthetic device 7012, walking may be problematic, since walkingmovement of the user's foot 7021, shown in FIG. 32, will cause the IMU7096 to sense orientation changes, which the device module 7017 will useto command movement of the prosthetic device 7012. However, walking maybe detected by providing an IMU 7096 having a third accelerometer 7114.Referring to FIG. 32, each of the accelerometers 7114 may be arranged tomeasure the acceleration in one of the X, Y or Z directions. Thus, whenthe user is substantially stationary, the vector sum of theaccelerations detected by each of the three accelerometers 7114 shouldbe substantially equal to 1G. When the Z axis is coincident with thedirection of gravity G, the accelerometer 7114 detecting acceleration inthe Z direction will detect the entire 1G acceleration due to gravity,since the accelerations in the X and Y directions will be substantiallyequal to zero. Now, referring to FIG. 40, when the user is stationary,but the direction of gravity G is not coincident with the Z axis, i.e.the user has moved their foot 7021 to command a pitch θ_(Pitch) and/orroll θ_(Roll), the vector sum of the accelerations Ax, Ay and Az in theX, Y and Z directions, respectively, will still equal 1G.

If the user begins to walk, the vector sum of the accelerations Ax Ayand Az detected by each of the three accelerometers 7114 will besubstantially greater than 1 G, since the act of walking will causeadditional acceleration, other than gravity, to be detected by the IMU7096. Therefore, referring to FIG. 41, once the IMU 7096 detects theaccelerations Ax, Ay and Az in S22, the vector sum of the accelerationsmay be compared to a walk detect limit in S23. In some embodiments, thewalk detect limit may be set at approximately 1.2G. If the vector sum ofthe accelerations is lower than the walk detect limit, in S24, thedevice module 7017, shown in FIGS. 31A and 31B, will command theprosthetic device 7012, shown in FIGS. 31A and 31B, in accordance withthe pitch θ_(Pitch), roll θ_(Roll) and/or yaw {dot over (θ)}_(Yaw)detected by the IMU 7096. However, if the walk detect limit is exceededby the vector sum of the accelerations, the device module 7017, shown inFIGS. 31A and 31B, will assume the user is walking and may alter thecontrol scheme for the prosthetic device 7012 in S25.

For example, in one embodiment of an altered control scheme when walkingis detected, the device module, shown in FIGS. 31A and 31B, filters outhigh accelerations from the pitch θ_(Pitch), roll θ_(Roll) and/or yaw{dot over (θ)}_(Yaw) signals generated by the IMU 7096, shown in FIG.32, when the vector sum of the accelerations Ax, Ay and Az is greaterthan the walk detect limit. Thus, when the user begins to walk,measurements having a value larger than the walk detect limit will notcommand movement of the prosthetic device 7012, shown in FIGS. 31A and31B. This walk detection embodiment is beneficial, as discussed above,because once the user is walking, the accelerations detected by theIMU(s) 7096 are large enough that the resulting signal may not becorrectly indicative of the user's intent. In some embodiments, thisembodiment of walk detection may be implemented where one or more IMUs7096 is worn on another area of the user, including, but not limited to,the residuum.

In another embodiment for a walk detection control scheme, when thecontrol apparatus 7010, shown in FIGS. 31A and 31B, senses a user iswalking, the device module 7017, shown in FIGS. 31A and 31B, may stopusing the signals generated by the IMU 7096, shown in FIG. 32, entirely,and instead switch to another sensor or signal input, e.g., EMG, todetermine user input intent. Additionally, in some embodiments, wherethe control apparatus 7010, shown in FIGS. 31A and 31B, detects that theuser is walking, the device module 7017, shown in FIGS. 31A and 31B, mayenter a standby mode, in which no commands are sent to the prostheticdevice 7012, shown in FIGS. 31A and 31B, as will be discussed in greaterdetail below. Entering standby mode both saves power and, also, preventsthe prosthetic device 7012, shown in FIGS. 31A and 31B, from executingpotentially erratic and unintended movement.

In some embodiments, the walk detect limit may need to be exceeded for apredetermined duration e.g., at least 6 seconds, before the controlapparatus 7010, shown in FIGS. 31A and 31B, implements a differentcommand after a predetermined amount of time the user is walking. Inthis embodiment, the device module 7017, shown in FIGS. 31A and 31B, maythen send a command signal to the prosthetic device 7012, shown in FIGS.31A and 31B, after the predetermined period has elapsed to place most ofthe controlled electronics (i.e., the electronics in the prostheticdevice 7012) into a sleep mode. When in sleep mode, if the controlapparatus 7010, shown in FIGS. 31A and 31B, determines that the user isno longer walking, which may be detected from the orientation signalsgenerated by the IMU 7096 indicating that the vector sum of theaccelerations Ax, Ay and Az is below the walk detect limit for thepredetermined period, the device module 7017, shown in FIGS. 31A and31B, may then turn the controlled electronics on again to allow normaloperation of the prosthetic device 7012, shown in FIGS. 31A and 31B.

In some embodiments, after the device module 7017 determines that theuser is no longer walking, the control system may enter standby modewhere it can be determined if the user has repositioned the IMU 7096and, if so, the IMU may be zeroed. Additionally, in some embodiments,when walk detection mode is entered, the device module 7017 may stay inits current mode of operation but ignore body input signals from the IMU7096 for a predefined walking time. Then, if the predefined walking timeis exceeded and the device module 7017 still detects that the user iswalking, the device module 7017 may enter standby mode to conservepower.

In another embodiment of a control scheme for when user walking has beendetected, the device module 7017, shown in FIGS. 31A and 31B, may ignoreonly the yaw signal {dot over (θ)}_(Yaw) when the walk detect limit isexceeded by the vector sum of the accelerations Ax, Ay and Az. Then,when the control apparatus 7010, shown in FIGS. 31A and 31B, determinesthe user has stopped walking, the device module 7017, shown in FIGS. 31Aand 31B, may begin using the yaw signal {dot over (θ)}_(Yaw) again. Insome embodiments, the device module 7017, shown in FIGS. 31A and 31B,re-zeros the yaw measurement when the large accelerations cease (i.e.,when the accelerations are below the walk detect limit).

In some embodiments, the control system of the present invention mayinclude a power free swing mode that may be activated automatically whenthe device module 7017, shown in FIGS. 31A and 31B, determines that theuser is walking. In power free swing mode, the device module 7017, shownin FIGS. 31A and 31B, may allow the joints of the prosthetic device7012, shown in FIGS. 31A and 31B, to freely move, so that the prostheticdevice 7012 may swing as a result of the user's walking movement. Powerfree swing mode provides the user with a more natural look while walkingwithout significantly increasing power consumption to do so.

In another embodiment, the control system may include a power swingmode, which also controls the prosthetic device 7012, shown in FIGS. 31Aand 31B, to swing as the user is walking. In various embodiments, theuser may pre-select or pre-program into the device module 7017, shown inFIGS. 31A and 31B, a default swing speed and then may increase ordecrease the speed during use. The user may select this mode and/or varythe speed by any of the various sensors 7018 and/or IMUs 7096 describedherein for user input. In some embodiments, the control apparatus 7010,shown in FIGS. 31A and 31B, in the power swing mode may additionallydetermine the stride length and rhythm of the user and regulate thepowered swing in response, to match the user's cadence.

Referring back to FIGS. 31A and 31B, as discussed above, the controlapparatus 7010 may control the prosthetic device 7012 in a variety ofcontrol modes to achieve different functionality from the prostheticdevice 7012. The use may enter and/or exit the different control modeswith signals from the various sensors 7018 and IMUs 7096 discussedabove. Additionally, some control modes may be entered automatically ifa preset condition is achieved, for example, entering power free swingmode or standby mode if the walk detect limit is exceeded, as discussedabove.

One control mode of the present invention may be a calibration mode thatthe user may enter once the user places the IMU(s) 7096, shown in FIG.32, on their foot or feet 7021, shown in FIG. 32. This calibration modemay negate any misalignments of the IMU 7096 on the user's foot 7021,for example, by setting the pitch and yaw offset angles discussed above.In some embodiments, the user may place the IMU(s) 7096 on their foot orfeet 7021 and then power the IMU(s) 7096 “on” to automatically enter thecalibration mode. Once in the calibration mode, the user may perform oneor more calibration movements with their foot or feet 7021, i.e., “towup”, “heel up”, “tilt side to side”, etc., to establish a baseline forthe range of motion of the user's foot or feet 7021, which may be used,for example, to set motion control gains such as gains k₁, k₂, k₃ and k₄discussed above. These calibration movements and their order ofperformance are for exemplary purposes only. In other embodiments,different calibration movements and/or a different order of performanceof calibration movements may be used, as should be understood by thoseskilled in the art. In various embodiments, the user may be required tocomplete a “range of motion” to establish a baseline and for the systemto establish the X, Y and Z axes. Other embodiments of the presentinvention may implement calibration modes as well. For instance, in theembodiment having the sensor grid 6084, shown in FIG. 8A, thecalibration mode may detect a current pressure profile of the user'sfoot so that changes in the pressure profile can then be detected tocontrol the prosthetic device 7012.

Referring to FIG. 42, as discussed above, one control mode of thepresent invention may be standby mode. The device module 7017, shown inFIGS. 31A and 31B, initiates standby mode upon receipt of a particularbody input signal from the sensor module 7015, shown in FIGS. 31A and31B, in S26. For instance, the body input signal may be generated byengaging a particular sensor 7018, by a specific orientation of one ormore IMUs 7096 or the like. Additionally, as discussed above, the signalmay be generated automatically, for example, if the control apparatus7010, shown in FIGS. 31A and 31B, detects that the user is walking. Uponentering standby mode, the prosthetic device 7012, shown in FIGS. 31Aand 31B, becomes frozen or locked in its current position in S27. Insome embodiments, this may include turning off actuators and turningbrakes and/or clutches on within the prosthetic device 7012. In someembodiments, while in standby mode, the device module 7017, shown inFIGS. 31A and 31B, does not send commands to the prosthetic device 7012,and therefore, does not command unintended movement of the prostheticdevice 7012. Standby mode is advantageous since it may allow the user tomaintain the prosthetic device 7012 in a desired position withoutsignificantly draining battery power.

The device module 7017, shown in FIGS. 31A and 31B, is maintained instandby mode until the device module 7017 receives an input signal inS28 indicating that a new control mode is to be entered. In someembodiments of the present invention, when standby mode is exited and anew control mode is entered, for example, bulk mode, finesse mode or thelike, the device module 7017 will send a zero command to the sensormodule 7015, which the sensor module 7015 may use to redefine its zeroposition or orientation to be the current position or orientation. Forexample, the device module 7017, shown in FIGS. 31A and 31B, may send azero command to the IMU 7096 of the sensor module 7015 by setting apitch offset, θ_(Offset) _(_) _(Pitch), and a roll offset, θ_(Offset)_(_) _(Roll). Thus, any orientation changes of the foot/feet 7021, shownin FIG. 32, occurring between the time the device module 7017, shown inFIGS. 31A and 31B, entered standby mode in S26 and the time that thedevice module 7017, shown in FIGS. 31A and 31B, exited standby mode inS28 are compensated for by the device module 7017. Once the controlsystem has been zeroed in S29, the device module 7017, shown in FIGS.31A and 31B, enters the new control mode in S30 and begins continuouslyreceiving data from the IMU(s) 7096 of the sensor module 7015, shown inFIGS. 31A and 31B, for controlling the prosthetic device 7012, shown inFIGS. 31A and 31B, in accordance with the new control mode. When in thenew control mode, the prosthetic device 7012 will not move while theuser's foot 7021 is in the zero position since the signals generated bythe IMU 7096 for the zero position will be interpreted as zero by thedevice module 7017. When the user's foot 7021 leaves the zero position,the device module 7017 uses the data signals from the IMU 7096 tocommand movement of the prosthetic device 7012, shown in FIGS. 31A and31B, based on the selected control mode.

The active zeroing process may be used in other embodiments and thus isnot limited to use with the IMU. Further, the zeroing process may bebeneficial for many reasons, including, but not limited to, where theuser moves from flat ground to a sloped ground, the controls mayinterpret this as a command. Thus, active zeroing eliminates this issuewhich may otherwise give ruse to unintended commands.

Referring to FIG. 43, bulk mode includes movement of the prostheticdevice 7012 into the general vicinity desired by the user. When in bulkmode, the signals from sensors 7018 and/or IMUs 7096 are used by thedevice module 7017, shown in FIGS. 31A and 31B, to move a prosthetic endpoint 7122 to a desired location 7124, i.e. a specific point in athree-dimensional space (x, y, and z components). For instance, in bulkmode, the signals from the IMUs 7096 and/or sensors 7014 may be used bythe device module 7017, shown in FIGS. 31A and 31B, to command shoulderabduction, M_(SA), about an abduction axis 7126, shoulder flexion,M_(SF), about a shoulder flexion axis 7128, humeral rotation, M_(HR),about a humeral rotation axis 7130 and elbow flexion, M_(EF), about anelbow flexion axis 7132. In this way, the user is able to move theprosthetic end point 7122 to the desired location 7124, withoutactuating a prosthetic wrist 7134 and hand 7136. Then, once theprosthetic end point 7122 has reached the desired location 7124, theuser may engage finesse mode to control the wrist 7134 and hand 7136, aswill be discussed in greater detail below.

In one embodiment, the pitch θ_(Pitch) roll θ_(Roll) and yaw {dot over(θ)}_(Yaw) detected by each IMU 7096, shown in FIG. 32, may correspondto movement of a specific joint of the prosthetic device 7012. Forexample, M_(Pitch) and M_(Roll), discussed above, may correspond toM_(SF) and M_(SA), respectively. Similarly, M′_(Pitch) and M′_(Roll) maycorrespond to M_(EF) and M_(HR) respectively. Thus, the user may movethe prosthetic end point 7122 to the desired location 7124 by pitchingand rolling each foot 7021, shown in FIG. 32, to move the appropriateprosthetic joints until the desired location 7124 is reached. Althoughdescribed as corresponding to specific joint movements for exemplarypurposes, it should be understood that M_(Pitch), M_(Roll), M_(Yaw),M′_(Pitch), M′_(Roll) and M′_(Yaw) may each be programmed in the devicemodule 7017, shown in FIGS. 31A and 31B, to correspond to any of thejoint movements, depending upon user preference. In the exemplaryembodiment discussed above, M_(Yaw), M′_(Yaw) and/or other sensors 7018,shown in FIGS. 31A and 31B, may be programmed to perform otherprosthetic functions such as mode switching or, alternatively, may notbe programmed to perform any function while the control system is inbulk mode.

Referring to FIG. 44, in another embodiment for end point control inbulk mode, the pitch θ_(Pitch), roll θ_(Yaw) and yaw {dot over(θ)}_(Yaw) signals generated by the IMUs 7096 may correspond directly tomovement of the prosthetic end point 7122. For example, M_(Pitch) may beprogrammed in the device module 7017, shown in FIGS. 31A and 31B, tocommand end point movement M_(Up) and M_(Down), M_(Roll) may beprogrammed in the device module 7017, shown in FIGS. 31A and 31B, tocommand end point movement M_(Right) and M_(Left), and M′_(Pitch) may beprogrammed in the device module 7017, shown in FIGS. 31A and 31B, tocommand M_(In) and M_(Out). Although described as corresponding tospecific directional movements of the prosthetic end point 7122 forexemplary purposes, it should be understood that M_(Pitch), M_(Roll),M_(Yaw), M′_(Pitch), M′_(Roll) and M′_(Yaw) may each be programmedwithin the device module 7017, shown in FIGS. 31A and 31B, to correspondto any of the directional movements, depending upon user preference.

In this embodiment, the device module 7017, shown in FIGS. 31A and 31B,commands shoulder abduction M_(SA) about the abduction axis 7126,shoulder flexion M_(SF) about the shoulder flexion axis 7128, humeralrotation M_(HR) about the humeral rotation axis 7130 and elbow flexionM_(EF) about the elbow flexion axis 7132 in accordance with a movementfunction to achieve the commended movement of the prosthetic end point7122, e.g. M_(Up) or M_(Down). This embodiment for control of theprosthetic end point 7122 may, in some embodiments, be preferable, sincethe user must only be concerned with directional movement of the endpoint 7122, rather than movement of individual joints to achieve thedesired end point movement.

As discussed above, the pitch θ_(Pitch), roll θ_(Roll) and yaw {dot over(θ)}_(Yaw) signals generated by the IMUs 7096, shown in FIG. 32, may bemapped either to the position or the velocity of the prosthetic endpoint 7122. In some embodiments, a faster or larger orientation changeor rate of change of an IMU 7096 may translate to a higher speed ofmovement of the prosthetic device 7012. Thus, in these embodiments, thefaster an orientation changes, the faster the prosthetic device 7012moves. However, in other embodiments, the control system may not includea speed variant, but rather the device module 7017, shown in FIGS. 31Aand 31B, may only command directional movement at preset speeds. In someembodiments, the larger movements by the prosthetic device 7012 may bequicker in speed for the first 80% of the desired movement, but maygradually slow for the last 20% of the desired movement to allow formore fine point control as the prosthetic end point 7122 reaches thedesired location 7124. For example, in some embodiments, when movingtoward the user's face, the prosthetic device 7012 may slow down whenapproaching or getting close to the face. This area of reduced speed maybe programmed directly into the end point control by defining a portionof the movement envelope near the user's face as a slow speed area. Thisfunctionality may beneficially make use of the prosthetic device 7012more comfortable and/or safe for the user. In some embodiments, thespeed of particular prosthetic joints of the prosthetic device 7012 mayalso be varied over the range of motion of the prosthetic joint, forexample, an elbow joint may slow down as it becomes more flexed.

Thus, in some embodiments, independent of user input, the control systemmay slow or quicken automatically based on preprogrammed speed controls.Thus, this may expand/improve the control resolution in one or moreareas of the envelope.

As discussed above, in some embodiments, the speed of the variousmovements of the prosthetic device 7012 may be controlled using the IMUs7096, and in some embodiments, it may be desirable to allow limits tothe speed of one or more types of movement of the prosthetic device 7012to be customized into the control system. Since some joints of theprosthetic device 7012 may need to actuate to a greater degree thanother joints to reach the desired location 7124, depending upon thelocation of the desired location 7124 relative to the prosthetic endpoint 7122, the velocity limit may be unique for that x, y, z locationof the desired location 7124. In some embodiments, the following methodmay be used by the device module 7017 to calculate the maximum speed.The steps include calculating the angles to reach the desired position;if any one of the angles exceeds the maximum difference allowed from thecurrent position, then the device module 7017 assumes the ratio of theangle is the same. Thus, if the difference required at X degree anglechange and the maximum allowed angle change is Xmax, the maximum vectorlength that is reachable is Xmax/X where X is the desired vector length.The new vector length may then be used as the desired input.

Referring to FIG. 45, the prosthetic device 7012 has a movement envelope7138 that includes a boundary 7140 that limits where the prosthetic endpoint 7122 of the prosthetic device 7012 is able to move relative to theuser. The movement envelope 7138 is dependent upon the length of theprosthetic device 7012, the length of the various segments of theprosthetic device 7012 and movement limitations of the joints of theprosthetic device 7012. The prosthetic end point 7122 may reach onlydesired locations 7124, shown in FIG. 40, that are on or within themovement envelope 7138. For example, with the prosthetic device 7012 inthe position shown, upward movement M_(Up) of the prosthetic end point7122 would require the prosthetic end point 7122 to move outside of themovement envelope 7138 and, therefore, cannot be executed. However,rather than simply preventing movement outside of the movement envelope7138 by stopping the prosthetic device 7012, the device module 7017,shown in FIGS. 31A and 31B, may instead follow the closest possiblemovement path by commanding movement along the boundary 7140 of themovement envelope 7138 that includes a component of the commandedmovement.

For example, as discussed above, with the prosthetic device 7012 in theposition shown, if the user commands upward movement M_(Up) of theprosthetic end point 7122, the end point 7122 would need to move outsideof the movement envelope 7138, which is not possible. Therefore, uponreceipt of a signal corresponding to upward movement M_(Up), the devicemodule 7017, shown in FIGS. 31A and 31B, may instead command movementM_(Boundary) _(_) _(Up) along the boundary 7140, which includes anupward component, but also an inward component. Similarly, if the usercommands outward movement M_(Out) of the prosthetic end point 7122, theend point 7122 would again need to move outside of the movement envelope7138, which is not possible. Therefore, upon receipt of a signalcorresponding to outward movement M_(Out), the device module 7017, shownin FIGS. 31A and 31B may instead command movement M_(Boundary) _(_)_(Out) along the boundary 7140, which includes an outward component, butalso a downward component.

Thus, this embodiment of the present invention is beneficial since itprevents the prosthetic device 7012 from becoming stuck in a positionalong the boundary 7140 of the movement envelope 7138 simply because theboundary 7140 has been reached. Although shown in two dimensions forsimplicity, it should be understood by those skilled in the art that themovement envelope 7138 will actually limit movement of the prostheticdevice 7012 in three dimensional space for a prosthetic device 7012 thatis able to move in three dimensions.

Referring to FIG. 46, in some embodiments, another IMU 7096 may be usedto measure the orientation of another part of the user's body, such asthe user's arm 7142 or hand 7144. In this embodiment, the device module7017, shown in FIGS. 31A and 31B, may enter a mimic mode to control bulkmovement of the prosthetic device 7012, in such a manner that the devicemodule 7017, shown in FIGS. 31A and 31B, will command the prostheticdevice 7012 to move the prosthetic end point 7122 to substantially mimicthe movement of the IMU 7096 on the user's arm 7142 or hand 7144. Thus,for example, if the user moves the IMU 7096 to the left, the devicemodule 7017 will move the prosthetic end point 7122 to the left. Inanother embodiment, using the IMU 7096 on the user's arm 7142 or hand7144, the device module 7017, shown in FIGS. 31A and 31B, may commandthe prosthetic device 7012 to move the prosthetic end point 7122 tosubstantially mirror the movement of the IMU 7096. Thus, for example, ifthe user moves the IMU 7096 to the right, the device module 7017 willmove the prosthetic end point 7122 to the left. Accordingly, althoughsome of the exemplary embodiments described herein referring to the useof a user's foot or feet to control a prosthetic device 7012, in otherembodiments, other body parts of the user may be used to provideorientation information.

Referring to FIG. 47, as discussed above, finesse movement relates tomanipulating an object and specifically relates to operating theprosthetic hand 7136 and the prosthetic wrist 7134. Operation of theprosthetic hand 7136 may include grip selection and/or actuation andincludes movement of a thumb structure 7148, an index structure 7150, amiddle structure 7152, a ring structure 7154 and/or a pinky structure7156, as will be discussed in greater detail below. Wrist operation mayinclude wrist rotation M_(WR) about a wrist rotation axis 7158, wristflexion M_(WF) about a wrist flexion axis 7160 and wrist deviationM_(WD) about a wrist deviation axis 7162. When in finesse mode, thesignals from sensors 7018 and/or IMUs 7096, shown in FIGS. 31A and 31B,are used by the device module 7017, shown in FIGS. 31A and 31B, tocommand the various wrist movements discussed above. For example,M_(Roll) may be used to command M_(WR), M_(Pitch) may be used to commandM_(WF) and M_(Yaw) may be used to command M_(WD). Additionally, in someembodiments wrist flexion and wrist deviation may be couple togethersuch that the prosthetic wrist 7134 follows a fixed path that includessome degree of wrist flexion M_(WF) and some degree of wrist deviationM_(WD). This embodiment is advantageous because it allows a singleinput, for example M_(Pitch), to move the prosthetic wrist 7134 alongthe fixed path, thereby controlling both wrist flexion M_(WF) and wristdeviation M_(WD) with the single input.

In some embodiments, where the prosthetic wrist 7134 is provided withthree degrees of freedom, i.e. wrist rotation M_(WR), wrist flexionM_(WF) and wrist deviation M_(WD), particular control schemes may beused with respect to the wrist flexion and deviation. For example, agravity based control scheme may be provided where signals from the IMUs7096, shown in FIGS. 31A and 31B, may control up/down and left/rightmovement of the wrist independent of the wrist rotation position, ratherthan individually controlling the wrist flexion M_(WF) and wristdeviation M_(WD). Additionally, the gravity based control scheme maymaintain an object held by the prosthetic hand 7136 at a specificorientation relative to gravity if the control mode is switched fromfinesse mode to bulk mode. This may be accomplished, for example, byproviding a sensor 7018 on/or within the prosthetic hand 7136 formeasuring the direction of the gravity vector G. The gravity basedcontrol scheme may be activated or deactivated in a manner similar tomode switching, for example by activating a switch or sensor. Thisgravity based control scheme may be particularly beneficial if the useris holding am object that could spill if inverted, such as a glass ofwater or the like.

In another control scheme, a single input may again be used by thedevice module 7017, shown in FIGS. 31A and 31B, to control both wristflexion M_(WF) and wrist deviation M_(WD), which may be of particularuse where only two degrees of freedom for control input are available.In such an embodiment, the controlled movement may be made orientationdependant. For example, the control signal may control wrist flexionM_(WF) when the prosthetic hand 7136 is facing palm down but may controlwrist deviation M_(WD) when the prosthetic hand 7136 is facing palmsideways.

In addition to control of the prosthetic wrist 7134, finesse mode alsoprovides for control of the prosthetic hand 7136. In particular, finessemode provides control for grip selection and actuation. Referring toFIG. 48, as used herein, a grip refers to the range of motion throughwhich the prosthetic hand 7136 passes from a fully open position 7164 toa fully closed position 7166. In some embodiments, the signals from theIMU 7096 and/or sensors 7018, shown in FIGS. 31A and 31B, allow the userto both fully or partially actuate each grip. For example, the user maymake the grip begin to close by pitching the IMU 7096 to generate thepitch signal θ_(Pitch). However, if the user returns the IMU 7096 to thezero position, the grip maintains its altered position. Then the usermay continue to close the grip by pitching the IMU 7096 again or,alternatively, may open the grip by pitching the IMU 7096 in theopposite direction.

In some embodiments, the device module 7017, shown in FIGS. 31A and 31B,includes a plurality of different preprogrammed grips that areselectable by the user. For example, the user may program specific inputsignals from the IMUs 7096 and/or sensors 7018, shown in FIGS. 31A and31B, to correspond to the specific grips. In one embodiment, the usermay set fore and aft pitch θ_(Pitch) and left and right roll θ_(Roll)detected by the IMU 7096 to correspond to four different hand grips. Inanother embodiment, as discussed above, one or more signals from theIMUs 7096 may be programmed to cycle forward or backward through a listof grips, thereby allowing the user to cycle through all of thepreprogrammed grips using a single IMU orientation signal. For example,left and right roll θ_(Roll) detected by the IMU 7096 may allow the userto cycle through the list of grips and fore and aft pitch θ_(Pitch) maythen allow the user to actuate, i.e. open and close, the selected grip.

In one embodiment, the device module 7017, shown in FIGS. 31A and 31B,includes six different preprogrammed grips that each close the thumbstructure 7148, index structure 7150, middle structure 7152, ringstructure 7154 and pinky structure 7156 in varying manners and withvarying trajectories. For example, referring to FIGS. 49A-49D, a “keygrip” 7168, which may also be referred to as a “lateral pinch grip”, mayfirst close the index structure 7150, middle structure 7152, ringstructure 7154 and pinky structure 7156, while moving the thumb outwardto a “thumbs up” position. Then, the thumb structure 7148 may be lowerto contact the index structure 7150. This key grip allows the user tohold an object (not shown) within the palm of the prosthetic hand 7136or to pinch an object (not shown) between the thumb structure 7148 andthe index structure 7150. Additionally, the user may halt actuationmidway through the key grip 7168, for example, to signal a “thumbs up.”The key grip 7168 also includes a dressing position within itstrajectory that may assist the user in putting the prosthetic handthrough a shirt or coat sleeve. The key grip 7168 also includes a handleposition within its trajectory in which the index structure 7150, middlestructure 7152, ring structure 7154 and pinky structure 7156 begin toclose to facilitate the grasping of a handle, such as the handle of abriefcase.

Referring to FIGS. 50A-50B, the control system may also be preprogrammedwith a power grip 7169. The power grip 7169 is similar to the key grip7168, shown in FIGS. 49A-49D, in that the index structure 7150, middlestructure 7152, ring structure 7154 and pinky structure 7156 are closedfirst, while the thumb structure 7148 is moved to be perpendicular tothe palm of the prosthetic hand 7136. Then, the thumb structure 7148 isclosed laterally along the index structure 7150 into a fist.

Referring to FIG. 51, the control system may also be preprogrammed witha tool grip 7170. The tool grip 7170 first closes the thumb structure7148, middle structure 7152, ring structure 7154 and pinky structure7156. Once closed, the index structure 7150 is then closed as well. Thisgrip is advantageous because it allows the user to grip a hand tool (notshown), such as a drill, or another similar object and then activate thecontrol for the hand tool, such as a drill trigger. Additionally, theuser may halt actuation midway through the grip to provide a handconfiguration useful for pointing at objects.

Referring to FIG. 52, the control system may also be preprogrammed witha chuck grip 7171 in which the orientation of the thumb structure 7148,index structure 7150 and middle structure 7152 is critical. The chuckgrip 7171 closes the thumb structure 7148 toward the base of the middlestructure 7152, while simultaneously closing the index structure 7150,middle structure 7152, ring structure 7154 and pinky structure 7156 tobring the thumb structure 7148 to the index structure 7150 and middlestructure 7152.

Referring to FIG. 53, the control system may also include apreprogrammed pinch open grip 7172. The pinch open grip 7172 leaves themiddle structure 7152, the ring structure 7154 and the pinky structure7156 open and brings the tip of the thumb structure 7148 and the tip ofthe index structure 7150 together to allow the user to pick up smallobjects (not shown).

Referring to FIG. 54, the control system may also include apreprogrammed pinch closed grip 7173. The pinch closed grip 7173 firstcloses the middle structure 7152, the ring structure 7154 and the pinkystructure 7156 and then brings the tip of the thumb structure 7148 andthe tip of the index structure 7150 together to allow the user to pickup small objects (not shown), while moving the unused finger structuresout of the way.

The position of the fingers/thumbs one to another in the various gripsmay be preprogrammed to maximize the effectiveness of the grips. Forexample, in the chuck grip and the power grip, the angle of orientationof the thumb with respect to the fingers may be changed in the controlsystem to optimize the grips. In some embodiments, the thumb positioningand various grip trajectories may be determined through one or more userstudies and/or user input to optimize one or more grips.

Although described as having six grips, it should be understood by thoseskilled in the art that the device module 7017 may be preprogrammed withessentially an infinite number of varying grips. In some embodiments,the infinite number of grips may be those mid-grips or grips formedwhile the hand is closing to one of the six grips described above.Additionally, although the signals from the IMU 7096 and the sensors7018 have been described as corresponding to specific joint movementsand grips for exemplary purposes, it should be understood thatM_(Pitch), M_(Roll), M_(Yaw), M′_(Pitch), M′_(Roll), M′_(Yaw) and thesignals from sensors 7018 may each be programmed in the device module7017 to correspond to any of the joint movements or grips, dependingupon user preference.

In some embodiments, it may be beneficial to provide tactile feedback tothe user, which may be a vibration, buzz or other, signaling to the userthat the hand is grasping. The tactile feedback may be generated by oneor more feedback sensors 14, shown in FIG. 1A, within the prosthetichand 7136 such as pressure sensors or force sensing resistors. In someembodiments, the tactile feedback may signal to the user the strength ofthe grip. In some embodiments, where the user is maintaining a steadygrip, i.e., no change in grip strength, the vibration or buzz may stopto signal to the user that the grip is maintaining a desired forcerather than changing the exerted force. In some embodiments, a/thechange in force/grip is indicated rather than a constant feedback wherethere is no change.

In some embodiments, the system may include user control of compliancefor appropriate circumstances. For example, but not limited to, wherethe user commands the thumb and index finger to close and the usercontinues to command the system to close even after the fingers arealready closed, in some embodiments, this may signal the system to backout compliance. This may provide a more forceful grip and the system maycontinue to increase the stiffness as user continues to commandincreased closing. Thus, in some circumstances, the system may lock outcompliance. This may be beneficial in making the fingers stiffer bymeasuring force and controlling the force (i.e., force control). Thiscontrol system may be additionally beneficial for it includes improvedinterpretation of user commands.

In some embodiments, for example, in lateral pinch grip, force feedbackmay be used. For example, but not limited to, where the index finger andthumb close on each other. At a predetermined point, the index fingercompliance bottoms out and the index finger position is maintained. Thethumb may continue to exert force onto the index finger until maximumtorque is being exerted onto the index finger.

Referring back to FIG. 32, in some embodiments, where the control systemincludes 2 IMUs 7096, one on each of the user's feet 7021, the controlsystem may include moving platform detection. For example, the devicemodule 7017, shown in FIGS. 31A and 31B, may disregard signals generatedby the IMUs 7096 when the signals indicative of pitch θ_(Pitch), rollθ_(Roll) and/or yaw {dot over (θ)}_(Yaw) generated by both IMUs 7096 aresubstantially identical. The device module 7017 will assume that thesubstantially identical signals generated by the two IMUs 7096 are dueto accelerations from the user's environment, for example if the user isriding in a vehicle such as car, a train, a plane or the like, ratherthan intended commands. It should be appreciated that in someembodiments of moving platform detection, a delta or differentialbetween the two IMUs 7096 may be used to command the prosthetic device7012. In some embodiments, this may be a selectable mode that the usermay elect during customization of the control apparatus 7010 so that theuser may activate the mode upon entering the vehicle or the like.

Referring back to FIGS. 31A and 31B, in some embodiments, the controlapparatus 7010 also includes a fail safe mode that the device module7017 will enter if a fail condition or error condition is detected fromfeedback sensors 7014. For example, the device module 7017 may enter thefail safe mode if power to the system goes out unexpectedly or ifcommunication with the IMUs 7096 is lost. In fail safe mode, theprosthetic device 7012 will remain in its current position and theprosthetic hand 7136 will open. The device module 7017 may turn theprosthetic actuators 7013 off and may engage brakes and/or clutches ofthe prosthetic device 7012. Other system failures or errors that willtrigger a failsafe may include, but are not limited to, sensor faults,motor or actuator faults (e.g., over current or over temperatureconditions), feedback position sensor signals out of a normal orexpected range, or a communication loss or communication errors betweenthe device module 7017 and the prosthetic device 7012.

In another embodiment of the present invention, the control apparatus7010 may include a computer mode that may be switched on and/or offusing any of the various IMUs 7096 and/or sensors 7014 described herein.When in the computer mode, body input signals from the IMUs 7096 and/orsensors 7014 may be used to control an associated external device, suchas movement of a mouse on a computer screen, movement of a car, ormovement of other similar remote-controlled devices.

In some embodiments, the control apparatus 7010 may be preprogrammedwith specific macros that may be executed in response to a particularbody input signal from one or more of the IMUs 7096 and/or sensors 7014.For instance, the specific macro may be a preprogrammed motion of theprosthetic device 7012 that is executed in response to a specificgesture, e.g., a double tap or a short tap of the foot 7021. In someembodiments, a macro may be programmed in real-time by the user, forexample, to “record” a motion and an associated instigator of thatmotion. Thus, in some embodiments, the user may “record” a performedmotion, and then, instigating the recording, the control apparatus 7010may repeat the motion. For example, when eating and/or drinking, theuser may find it helpful to record the specific motion and easily repeatthe motion by creating a macro.

As discussed above, the device module 7017 includes a prostheticcontroller 7027 that is in wireless or wired communication with theprosthetic device 7012. In the exemplary embodiment, the prostheticdevice 7012 may regularly communicate actuator status information, e.g.position information, to the device module 7017 and listens for, andexpects to receive, commands at regular intervals from the device module7017. In some embodiments, if the prosthetic device 7012 does notreceive commands from the device module 7017 within a pre-set amount oftime, the prosthetic device 7012 may shut down its actuators 7013 andturn on brakes.

As discussed above, in certain embodiments of the present invention,there are a number of modes in the control system. Thus, in theseembodiments, the pitch θ_(Pitch), roll θ_(Roll) and yaw {dot over(θ)}_(Yaw) signals from the IMU(s) 7096 may be translated to controldifferent functions for each of the different modes. For example, thepitch signal θ_(Pitch) from one IMU 7096 may control left/right movementof the prosthetic end point 7122, shown in FIG. 40, in bulk mode and maycontrol grip opening/closing movement in finesse mode. Additionally, ifthe control system includes other control modes, the pitch signalθ_(Pitch) from the IMU 7096 may also control other functions within eachof those control modes.

In some embodiments, mode switching between the various control modesmay be accomplished through an electrical switch or with one or moresensors 7018. Additionally, mode switching may be provided by moving theuser's foot 7021 in a specific gesture that has been preprogrammed to berecognized by the device module 7017, e.g., a double tap or a short tapof the foot 7021. In other embodiments, mode switching may beaccomplished using any other type of switch or signal, including, butnot limited to, a myoelectric switch, such as those known in the art(e.g., in some embodiments of control of a prosthetic arm, for transhumeral users, the tricep and/or bicep and/or pectoral muscles may beused, or, for transradial users, forearm muscles may be used).

In the exemplary embodiment, the switch used to switch between thecontrol modes may also be used to switch the prosthetic device 7012 andcontrol apparatus 7010 from an “on” state to an “off” state.Additionally, in some embodiments, a short tap of the foot 7021 mayswitch the mode (i.e., from bulk mode to finesse mode and visa versa)and a double tap of the foot 7021 may switch the system from the “on”stated to the “off” state. It should be understood that the double tapcould be done elsewhere on the user's body, i.e. in locations other thanthe foot, with another IMU 7096. In some embodiments, switching thecontrol apparatus 7010 to the “off” state maintains the current positionof the prosthetic device 7012, including the prosthetic hand 7136.

Still referring to FIGS. 31A and 31B, as discussed above, the controlapparatus 7010 is in some embodiments, customized to the user. Forinstance, the correspondence between each of the signals generated bythe IMUs 7096 and the control commands sent by the device module 7017 tothe prosthetic device 7012 may be customized. In one embodiment,customized correspondence may be mapped in a matrix that is uploaded todevice module 7017 of the control apparatus 7010. Then, when the devicemodule 7017 receives orientation signals from the IMUs 7096, the devicemodule 7017 is able to map the signal to the correct control command tobe sent to the prosthetic device 7012.

In embodiments where the user is using one IMU 7096 per foot 7021,movements of each foot 7021 may be linked to or mapped to correspondingmovements or types of movements for each mode of the prosthetic, i.e.bulk mode and finesse mode. In the exemplary embodiment, thecustomization allows assignment for each orientation signal generatedfrom each IMU 7096 to include specifically whether the user desires: 1)the particular position of the IMU 7096 to control the position of oneof the prosthetic joints; 2) the particular position of the IMU 7096 tocontrol the velocity of one of the prosthetic joints; or 3) the rate ofchange of the IMU 7096 to control the velocity of one of the prostheticjoints. It should be noted that in various embodiments, in addition tothe IMUs 7096, other inputs, e.g., sensors 7018 or EMG, may also bemapped to corresponding controls of the prosthetic device 7012. Forexample, EMG signals, in some embodiments, may also be mapped tomovements and types of movements of the prosthetic device 7012.

As discussed above, the IMU signals may be assigned to control movementsin each of the different control modes. Additionally, as discussedabove, some sensor signals or IMU signals may be assigned to controlmode switching (i.e., various foot taps may turn the system “on” or“off” and may switch the mode between bulk mode and finesse mode). Thismode and on/off switching is also customizable in the exemplaryembodiment. Additionally, some sensor and IMU signals may be assigned totoggle forward or backward through a list, i.e., to toggling throughvarious grips of the prosthetic hand 7136, shown in FIG. 43. Thus, thepresent invention allows for full customization between the variousinput devices of the sensor modules 7015 and the output that is to becommanded by the device module 7017.

For example, in the embodiment including IMUs 7096 on both feet 7021 ofthe user, the pitch θ_(Pitch) from the user's right foot 7021 may beassigned “elbow flex” in bulk mode and “wrist flex” in finesse mode. Theroll θ_(Roll) from the user's right foot 7021 may be assigned “humeralrotate” in bulk mode and “wrist rotate” in finesse mode. The pitchθ′_(Pitch) from the user's left foot 7021 may be assigned to toggleforward and backward through the grip options in finesse mode. The rollθ′_(Roll) from the user's left foot may be assigned to open and closethe prosthetic hand 7136 in finesse mode. Depending on the prostheticdevice 7012, there may be various IMU signals translating by the devicemodule 7017 to various control commands for both bulk mode and finessemode. Additionally, in embodiments having a single IMU 7096, theorientation signals from the single IMU 7096 will be assigned thevarious control commands.

As discussed above, the IMUs 7096 may be placed elsewhere on the user,for example, the shoulder, residuum, knee or lower leg. Referring toFIGS. 55-57, in some embodiments, rather than the IMU 7096, whichdetermines the orientation of the user's foot 7021 relative to gravity,the orientation of the user's foot 7021 may instead be determined bymeasuring the distance from the bottom of the user's foot 7021 to theground 7174. In some embodiments, this measurement may be determinedusing optical sensors 7176 located at a plurality of locations on thebottom of the user's foot 7021. For example, one optical sensor 7176 maybe located at the heel 7178 of the foot and two optical sensors 7176 maybe located along opposite edges of the foot in the in a toe region 7180.These optical sensors 7176 measure the distance between the user's footand the surface 7174 the user is on. For example, when the user pitchestheir foot upwards, as seen in FIG. 56, the sensor may determine thepitch distance 7182 and the sensor CPU 7019 may compute the angleθ_(Pitch) of the bottom of the user's foot 7021 to the surface 7174.Similarly, if the user rolls their foot sideways, as seen in FIG. 57,the sensor may determine the roll distance 7184 and the sensor CPU 7019may compute the angle θ_(Roll) of the bottom of the user's foot 7021 tothe surface 7174.

Various control systems have been described herein including those toimpart end-point control onto a prosthetic device 7012. Although theexemplary embodiments of the present invention discuss control systemsfor users that are shoulder disarticulation amputees, the currentmethods and systems may be broken down for use with prosthetic devicesfor trans-humerus and trans-radial amputees. For example, if the user'sarm has humeral rotation, the bulk movement may be simplified to includeonly elbow flexion. Similarly, for trans-radial amputees, bulk movementmay be provided entirely by the user's arm, with the control systemproviding only finesse movement. Thus, depending on the user's degree ofamputation, the bulk mode provided by the control system may be changedor removed entirely, such that some embodiments of the present inventionwill provide both bulk and finesse modes, other embodiments will provideonly the finesse mode and still other embodiments will provide partialbulk control along with the finesse mode.

As discussed above in connection with FIG. 1A, the feedback sensors 14of the prosthetic device 12 send signals to the device module 17 thatthe device module may use to command the actuators 13 of the prostheticdevice 12. Additionally, the device module 17 may also advantageouslystore data relating to the usage of the prosthetic device 12 to allowthe control system 10 to be tailored to the particular user and/or toallow a technician to identify portions of the prosthetic device 12 thatmay be improved. Referring to FIG. 58, in some embodiments, the devicemodule 17, shown in FIG. 1A, is programmed with a set of categories foreach feedback sensor 14, shown in FIG. 1A, in S32, spanning the totalrange of possible signals received from the feedback sensor 14, shown inFIG. 1A. For example, if the feedback sensor 14, shown in FIG. 1A, ismeasuring rotational position of a prosthetic joint that is capable ofrotating from a zero degree (0°) position to a ninety degree (90°)position, the set of categories may be, for instance, zero degrees tofifteen degrees) (0°-15°, fifteen degrees to thirty degrees) (15°-30°,thirty degrees to forty-five degrees) (30°-45°, forty-five degrees tosixty degrees) (45°-60°, sixty degrees to seventy-five degrees) (60°-75°and seventy-five degrees to ninety degrees) (75°-90°. As should beunderstood by those skilled in the art, the number of categories in theset or categories and/or the size of each category within the set ofcategories may be varied for each feedback sensor 14, shown in FIG. 1A,depending upon the desired measurement precision.

The prosthetic device 12, shown in FIG. 1A, is then operated by the userin S34. While the prosthetic device 12, shown in FIG. 1A, is inoperation, feedback signals from the feedback sensors 14, shown in FIG.1A, are transmitted to, and received by, the device module 17, shown inFIG. 1A, in S36. The device module 17, shown in FIG. 1A, identifieswhich category of the set of categories that the feedback signal fallsinto in S38 and records the total duration of time that the at least onefeedback signal is in the identified category in S40. This process maycontinue until it is evaluated in S42 that the prosthetic device 12,shown in FIG. 1A, is no longer in operation. Once the device is nolonger in operation, the recorded duration data may be evaluated by thetechnician in S44.

In particular, the technician may generate various plots to evaluate thetotal time that the particular feedback signal was in each particularcategory. For instance, referring to FIG. 59, a wrist rotationalposition histogram 186 may be generated for a prosthetic wrist joint toevaluate an accumulated time 188 that the wrist joint was in eachposition category 190. Similarly, referring to FIGS. 60 and 61, thetechnician may form histograms 186 showing accumulated time 188 versecategories 190 for a variety of other feedback signals from a variety ofother feedback sensors 14, including joint velocity, joint loading,actuator current, actuator torque, battery temperature, or the like. Forinstance, the prosthetic hand assembly may include position and/or forcesensors on various fingers, allowing usage of the various gripsdiscussed above to be evaluated.

Additionally, although described in terms of a feedback sensors, thedevice module 17, shown in FIG. 1A, may also collect durational datawith regard to the time in which the prosthetic device 12, shown in FIG.1A, is controlled in each of the various prosthetic control modesdescribed herein.

This durational data collected by the device module 17, shown in FIG.1A, allows the technician to configure the prosthetic device 12, shownin FIG. 1A, for each particular user. For example, the technician mayprogram the most intuitive body input signals from the IMUs 7096 andsensor 7018, shown in FIG. 31A, to be used by the device module 7017,shown in FIG. 31A, to command the most used prosthetic motions.Similarly, the technician may program lesser used control modes to becommanded by less intuitive body input signals. In some cases, thecontrol system 10 for particular users may even be customized to removecontrol modes that are not used by those particular users.

In addition to allowing for customization of the control system 10,shown in FIG. 1A, the durational data may also be used to identify areasfor improvement of the prosthetic device itself. For instance, thedurational data may indicate particular joints of the prosthetic devicethat are underpowered, allowing them to be redesigned to provideadditional power, or overpowered, allowing them to be redesigned toreduce weight of the prosthetic device 12, shown in FIG. 1A. Similarly,for overpowered joints, the prosthetic device 12, shown in FIG. 1A, maybe redesigned to reduce the battery power used by those joints toimprove batter life. Additionally, the durational data based on batterycurrent, temperature and/or capacity may aid in the selection of abetter battery for the prosthetic device 12, shown in FIG. 1A. Thus, thecollection of durational data by the device module 17, shown in FIG. 1A,advantageously allows for both customization and improvement of theprosthetic device 12, shown in FIG. 1A.

As is discussed herein, various embodiments of device modules and sensormodules as well as the control apparatus have been described. Thecontrol apparatus may be used to control a prosthetic device using oneor more sensors. With respect to switch-based sensors, including but notlimited to, foot sensors and/or joysticks, etc., these sensors includean application of force onto the switch-based sensor and a reactionforce. Thus, for example, with respect to foot sensors, the applicationof force by the user to a specific area (e.g., location of the sensor)may be necessary for the sensor to receive the signal from the user.However, where there is no reaction force, e.g., when the user's shoe isnot against a surface, the sensor may not receive the signal. Further,as sensors may require the application of force on a particular point toreceive a signal, this may present additional difficulties. Also, theapplication of force to a sensor may contribute to soreness or otherirritation imparted onto the user by the repetition of force applicationon a particular point of the user's foot and/or other body area.

Additionally, although switch-based sensors may be used, in someembodiments, it may be difficult for the sensors to receive signalsrelated to multiple axes at the same time. In some embodiments, formultiple-axis movement, the sensors may require receipt of multipleinputs regarding various axes. In some embodiments, these multipleinputs may be coordinated by the user, and in some embodiments, multipleinputs may be received by the sensor and then coordinated by the controlsystem for a determination of intended/desired multiple axis movement(i.e., user command). This may contribute to less control resolutionand/or may contribute to difficulty in use.

Thus, it may be desirable to use at least one non switch-based sensor toreceive user input regarding desired/intended motion of a prostheticand/or other device. As discussed herein, an IMU may be used. However,other non-switch based sensors may be used in various embodiments. Insome embodiments, the non-switch based sensor may include receivinginput from the user regarding desired/intended motion of a prosthetic ofother device. In some embodiments, the non-switch based sensor is notreliant on force application (and reaction force) and/or position of thesensor. These non-switch based sensors may be beneficial for manyreasons, including but not limited to, one or more of the following. Thesensor may sense motion without the application of force. The sensor mayreceive multiple axis input with a single motion (rather than multiple,coordinated motions). The sensor may be placed anywhere and receiveindication of intended/desired movement through motion. Once placed in aposition (e.g., anywhere, for example, but not limited to, connecteddirectly or indirectly on the user) a position may be “zeroed” and thus,change in position, including but not limited to, rate of change ofposition and/or the derivative of the position (acceleration) and/ordistance covered by the change in position, may be used as inputs to thecontrol system. In some embodiments, a sensor that may indicate thedesired and/or intended direction and/or speed and/or position of theprosthetic and/or other device may be the input. In some embodiments,the sensor may include, but is not limited to, one or more EEG or EMGsignal(s) from the user and/or one or accelerometers and/or one or moregyroscopes. In various embodiments, the sensor may be any sensor thatmeets one or more of these stated functions and/or benefits.

Therefore, various embodiments of the control apparatus includedirectional and proportional control of a prosthetic device and/or otherdevice without reliance on one or more switches and/or reaction forceand without concern for position of the sensor (i.e., the sensor may be“zeroed” or “nulled out” and/or position is not indicative (e.g., EEGand/or EMG signal)). Further, in some embodiments, rate of change etc.may be used by the system/apparatus for proportional and/or directionalcontrol so that, in various embodiments, input from the sensor may beused to command a device, including but not limited to, a prostheticdevice.

While the principles of the invention have been described herein, it isto be understood by those skilled in the art that this description ismade only by way of example and not as a limitation as to the scope ofthe invention. Other embodiments are contemplated within the scope ofthe present invention in addition to the exemplary embodiments shown anddescribed herein. Modifications and substitutions by one of ordinaryskill in the art are considered to be within the scope of the presentinvention.

What is claimed is:
 1. A control system for a prosthetic device comprising: at least one sensor module comprising at least one inertial measurement unit adapted to detect user input and to generate at least one body input signal based on the detected user input, the at least one inertial measurement unit comprising at least one accelerometer; at least one device module including a central processing unit and having at least one prosthetic control mode, the at least one device module being in communication with the at least one sensor module; and at least one actuator of the prosthetic device configured to receive commands from the at least one device module, wherein the at least one device module receives the at least one body input signal from the at least one sensor module and commands the at least one actuator in accordance with the at least one body input signal, wherein the at least one sensor module sends body input signals to the at least one device module according to the accelerations detected by the at least one accelerometer and the at least one device module detects user walking based at least on the body input signals, and wherein the at least one device module stops sending commands to the at least one actuator of the prosthetic device when user walking is detected.
 2. The control system according to claim 1 further comprising wherein the at least one device module commands the at least one actuator in accordance with the at least one body input signal and a current control mode of the at least one prosthetic control mode.
 3. The control system according to claim 1 further comprising wherein the inertial measurement unit comprises at least one accelerometer that is configured to detect accelerations in three perpendicular axes.
 4. The control system according to claim 1 further comprising wherein the inertial measurement unit includes at least three accelerometers, each accelerometer adapted to detect acceleration in a single axis, the at least three accelerometers oriented with respect to one another to detect accelerations in three perpendicular axes.
 5. The control system according to claim 1 further comprising wherein the at least one prosthetic control mode includes a bulk mode controlling movement of an end point of the prosthetic device within a movement envelope.
 6. The control system according to claim 5 further comprising wherein the at least one device module commands movement of the end point of the prosthetic device within the movement envelope in accordance with the at least one body input signal, the at least one body input signal being indicative of a desired directional movement of the end point of the prosthetic device.
 7. The control system according to claim 6 further comprising wherein the at least one device module commands movement of the end point of the prosthetic device along a boundary of the movement envelope when the desired directional movement of the end point of the prosthetic device required by the at least one body input signal would require movement of the end point of the prosthetic device outside of the movement envelope, the movement along the boundary of the movement envelope including at least a first directional component of the desired directional movement commanded by the at least one body input signal and at least a second directional component different from the desired directional movement commanded by the at least one body input signal.
 8. The control system according to claim 1 further comprising wherein the at least one prosthetic control mode includes a finesse mode controlling movement of a prosthetic hand of the prosthetic device.
 9. The control system according to claim 8 further comprising wherein the finesse mode additionally controls movement of a prosthetic wrist of the prosthetic device.
 10. The control system according to claim 8 further comprising wherein the finesse mode includes a plurality of grips controlling movement of the prosthetic hand.
 11. A control system for a prosthetic device comprising: at least one sensor module comprising at least one inertial measurement unit adapted to detect user input and to generate at least one body motion input signal based on the detected user input, the at least one inertial measurement unit comprising at least one accelerometer, the at least one sensor module being zeroed at a first orientation; at least one device module including a central processing unit and having at least one control mode, the at least one device module being in communication with the at least one sensor module; and at least one actuator of the prosthetic device configured to receive commands from the at least one device module, wherein the at least one device module receives the at least one body motion input signal from the at least one sensor module and determines a change in orientation based on the at least one body motion input signal, and wherein the at least one device module commands the at least one actuator in accordance with the determined change in orientation, wherein the at least one sensor module sends body input signals to the at least one device module according to the accelerations detected by the at least one accelerometer and the at least one device module detects user walking based at least on the body input signals, and wherein the at least one device module stops sending commands to the at least one actuator of the prosthetic device when user walking is detected.
 12. The control system according to claim 11 further comprising wherein the at least one device module determines the change in orientation between the first orientation and a second orientation and wherein the at least one device module commands the at least one actuator to actuate in accordance with the determined change in orientation.
 13. The control system according to claim 11 further comprising wherein the at least one device module determines a rate of change in orientation between the first orientation and a second orientation and wherein the at least one device module commands the at least one actuator to actuate in accordance with the determined rate of hange in orientation. 